Magic Lantern Forum

Experimental builds (WIP) => crop_rec and derived builds => Topic started by: reddeercity on April 06, 2017, 12:22:27 AM

Title: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 06, 2017, 12:22:27 AM
Full width 5.6k 16x9 A.R. , 4K Anamorphic 3x1, 1x3,  UHD 3K Plus 10-12bit Full HD ,
this is the latest build as things process I'll keep updating this post

See below short video on how to enable 48fps 3x3 preset

October 03 2020
Experimental build:MLV_Play.Mo and Raw_Twk.Mo Updated for Crop_Rec.Mo 4k3x1-3k1x1-2.8k1x1-48pfhd3x3
5D2_mlv_play+raw_twk_10-3-2020_8.18pm.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/5D2_mlv_playraw_twk_10-3-2020_8.18pm.zip)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FggbjTJt%2F5d2-mlv-play-10-3-20-small.png&hash=1afec087ce5bc22d78f79effdd28d7d2) (https://imgbb.com/)

Fixed problems with raw_twk & mlv_play modules on my D4/5D2 4k Crop_Rec branch .
Tested it in 3x3(1880x1258) , crop_rec 3.5k centered preset(2784x1160) & 3x1 4k anamorphic (4096x590)
there's all view ok in camera , except for 3x1 4k there noise/snow on the bottom half of the screen .
(the anamorphic part of the code is not vertically stretching the image need to look further in the code) but still viewable .
Make sure you load both module's in the zip file , that's it . when you what to view a file just press the > play button 


September 3 2020
Experimental build:50D eXperimental-crop-rec_3.7k_24fps
magiclantern-eXperimental-crop-rec_3.7k-1x1.11.30pm.2020Aug16.50D109.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/magiclantern-eXperimental-crop-rec_3.7k-1x1.11.30pm.2020Aug16.50D109.zip)
Here is a stable Crop_Rec build for the 50D , so no more adtg_gui crop_rec needed.
There only one preset at the moment , as these's problem with cmos & adtg hooks which i'm working on .
Loads just like the 5D2 crop_rec module , the 50D Crop_Rec resolution is 3760x1080 @ 23.976fps
50D related info for crop_rec https://www.magiclantern.fm/forum/index.php?topic=19336.msg229944#msg229944
 
July 20 2020
Experimental build:Crop_Rec.Mo Updated for 4k3x1-3k1x1-2.8k1x1-48pfhd3x3
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5D2-4k1x3-3k1x1-2.8k1x1-48pfhd3x3_7-10-2020_8.42pm_crop_rec.zip
New Google Drive updated download link
5D2-4k3x1-3k1x1-2.8k1x1-48pfhd3x3_7-10-2020_8.42pm_crop_rec.zip (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
5D2-4k1x3-3k1x1-2.8k1x1-48pfhd3x3_7-10-2020_8.42pm_crop_rec.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/5D2-4k1x3-3k1x1-2.8k1x1-48pfhd3x3_7-10-2020_8.42pm_crop_rec.zip)

Updated the Crop_Rec Module for multiplies of 3xcrop_mode/5xZoom presets .
Now there is 3k 1:1 (3016x1080 23.98fps) & Centered 3.5k 1:1 (2784x1160 24fps) in the same build plus 4k(3x1) & FHD 3x3 48p (1880x774)
With the new updated mlv_rec. module you get 8 more pixel in 3k & 24 more pixel in 48p 3x3 when use together .
Short Video tutorial on how to enable the new crop_rec presets in the latest update - (shot my 50D in 1x3 (1376x1742 @24fps ->4128x1742 from mlv. app)  ;)


July 5 2020
Experimental build:MLV_Rec.Mo Updated for 1880 (3x3) 2152(5xZoom)
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1880_3x3-2152_5xZoom-5d2_7-5-2020-5.59pm-mlv_rec.zip
New Google Drive Download Link
1880_3x3-2152_5xZoom-5d2_7-5-2020-5.59pm-mlv_rec.zip (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
1880_3x3-2152_5xZoom-5d2_7-5-2020-5.59pm-mlv_rec.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/1880_3x3-2152_5xZoom-5d2_7-5-2020-5.59pm-mlv_rec.zip)

In Canon default  resolution 3x3 you gain 24 more pixels (1856->1880) & in 3xcrop_mode(5xZoom) there a gain of 8 pixels (2144->2152)
Will load on top of the Crop_Rec eXperimental builds , did not test with the nightly builds
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FJvnTghP%2FVRAM103-small.png&hash=bf14c7dfb4a01375ff1e5526ea3a7c2d) (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F0cWP49X%2FVRAM104-5x-Zoom-small.png&hash=fee840fcccb35d03da13727b82bd0fce) (https://imgbb.com/)


Jan 5 2020
Experimental build:5632x3124 @ 3.8fps
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/crop_rec-5632x3124_3.8fps-5D2-eXperimental.9.42pm.2020Jan05.5D2212.zip
New Google Drive download link
crop_rec-5632x3124_3.8fps-5D2-eXperimental.9.42pm.2020Jan05.5D2212.zip (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
crop_rec-5632x3124_3.8fps-5D2-eXperimental.9.42pm.2020Jan05.5D2212.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/crop_rec-5632x3124_3.8fps-5D2-eXperimental.9.42pm.2020Jan05.5D2212.zip)

This a 1:1 pixel 5632x3124 @ 3.8fps , Very eXperimental you need at least a 1066x CFcard for this .
Read the discussion in these posts
https://www.magiclantern.fm/forum/index.php?topic=19336.msg223937#msg223937
https://www.magiclantern.fm/forum/index.php?topic=19336.msg224036#msg224036

Update: Feb 10 2020             New video sample of the updated 16x9 1:1 5632x3124 3.8fps centered preset , 6min @ 3.8fps -> 1 min 24fps in a hyper lapse


Here is a sample from the 5.6k preset (5632x2356 4.22fps) **note** this the first draft of this preset , I have updated since.
Shot this Jan-3-2020 , at sunset (4:30pm mst) , use my ef24-70mm 2.8L canon lens @ 24mm
F8 , ISO 100 , 1/5th second and because it was a sunset I had to use a Variable ND filter and stop it down 4 stops
I processed the 10bit cdng's thought Adobe After Effect CS6 with ACR ,I didn't spend much time on grading , just the basic .
I capture 7:00 min's continuously to a 32GB Lexar 1066x CF Card ,
I set the A.E. project to 24fps and that made the time condensed to about 1:15 seconds from 7:00 min's .
Watch it @ 5K (2880p) for the best viewing .
 


Dec 31 2019
Experimental build: 4k(4096) 3x1 Anamorphic ,  2784x1160 1x1 @ 24.00fps & FHD 3x3 (1856x774) @ 48p
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/crop_rec-4k3x1-2.8k24p_3x3-48p-all-centered_8.18pm-5D2-eXperimental.2019Dec31.5D2212.zip
New Google Drive Download link
crop_rec-4k3x1-2.8k24p_3x3-48p-all-centered_8.18pm-5D2-eXperimental.2019Dec31.5D2212.zip (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
crop_rec-4k3x1-2.8k24p_3x3-48p-all-centered_8.18pm-5D2-eXperimental.2019Dec31.5D2212.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/crop_rec-4k3x1-2.8k24p_3x3-48p-all-centered_8.18pm-5D2-eXperimental.2019Dec31.5D2212.zip)

In proved 4k-3x1 HDMI preview at 480p & changed frame rate to 24.00fps , reconfigured 2784x1160 for true 24.00 fps & realtime preview,
(1856x774  2:40 A.R. @ 48.028fps) write speed is 82MB/s so close to continuous
reduced the rolling shutter in 48p from 29.7ms to 17.8ms  :D 

August 21 2019
Experimental build: Dual ISO Video Module
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dual_iso_video-updates_8-21-2019.zip
New Google Download download link
5d2_dual_iso_video_updated_21-8-2020 (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
5d2_dual_iso_video_updated_21-8-2020.rar (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/5d2_dual_iso_video_updated_21-8-2020.rar)

Works with my crop_rec builds (not too sure about the nightly builds didn't test them) at 3x3 FHD(1856x1248) & 5xZoom(2144x1074) (3x crop_mode) @ 10bit raw ,
I didn't check it with the Crop_Rec Preset , so if you do try it with Crop_Rec it may freeze you cam and may need I battery pull .

Oct. 22 2019
Experimental build: 50D ADTG_GUI Module Crop_Rec experiment 3744x1080 @ 23.976 fps
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/adtg-gui_crop_rec-3.7k-50D-eXperimental.2019Oct01.50D109.zip
New Google Download Link
adtg-gui_crop_rec-3.7k-50D-eXperimental.2019Oct01.50D109.zip (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
adtg-gui_crop_rec-3.7k-50D-eXperimental.2019Oct01.50D109.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/adtg-gui_crop_rec-3.7k-50D-eXperimental.2019Oct01.50D109.zip)

Hey 50D users I've started to port 50D to crop_rec , first step was to code a simple preset in ADTG_GUI.mo where a1ex first tested the 5d3 code.
Reasons why I put the 50D crop_rec in adtg_gui is to make sure it work or course .

How to get started:
Load mlv_rec.mo & adtg_gui.mo
Enable adtg_gui go to the advance menu and enable ENGIO Registers

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FVvfN2g6%2FVRAM0-small.png&hash=4589146cad995d7072044f016096af03) (https://imgbb.com/)

Press half shutter to exit to liveview then press the play button (make sure to have a cr2 there to view ,this refreshes liveview & enables the other 1500 reg's)
go back to the advance tab in the adtg_gui.mo and now enable the 3k 1:1 50D preset

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F48Wwqzk%2FVRAM2-small.png&hash=fb5a351edc6c965998e40b3ce0476cb7) (https://imgbb.com/)

Press half shutter to exit to liveview , then you may have to cycle thought the 5xzoom button ( but you should only have to press it once)
Make sure you set the shutter to as close to 1/48th of second as you can , I was able to set it to 1/49th and when the preset is enabled it will
go to 1/48th of a second  for 23.9876 fps @ 3744x1080

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F6XXtmBW%2FVRAM3-small.png&hash=b9b22f35dcd1f35fadcc335cc9995d36) (https://imgbb.com/)

Don't worry about the line on the right had side , there will not be in the raw image (this is the same thing that happen with the 5D2 , still working on a fix )
To make sure everything gets loaded , you can check the memory patches in the debug menu you should have 7 like this

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FxJFSvcv%2FVRAM4-small.png&hash=b2ffe9a65707113ae556557d9d412a6d) (https://imgbb.com/)

On a Lexar 1066x 64GB CF card was getting around 10 second in 10bit .

This is very Bleeding Edge Stuff , so expect lockups , cam liveview freezes , etc. .... if it brakes you get to keep both halves  :P



New Update: November 14/2019
3008x1080 @ 23.98 fps Updated Corrected Horizontal Centering
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-crop_rec-3k_Updated_Center_4.20pm-5D2-eXperimental.2019Nov14.5D2212.zip
New Google Drive Download Link
magiclantern-crop_rec-3k_Updated_Center_4.20pm-5D2-eXperimental.2019Nov14.5D2212.zip (https://drive.google.com/file/d/)

New Updated Bitbucket Down Load Link(8-27-2020)
magiclantern-crop_rec-3k_Updated_Center_4.20pm-5D2-eXperimental.2019Nov14.5D2212.zip (https://bitbucket.org/reddeercity/crop_rec_5d2_50d/downloads/magiclantern-crop_rec-3k_Updated_Center_4.20pm-5D2-eXperimental.2019Nov14.5D2212.zip)

Details lower down on this post

*******************************************************************************************************************

So now you have stable 10-12bit in 1:1 Full HD with 3k crop_rec , more usable now .
To come full hd 48fps or maybe 50 & Full resolution 5.6k @4 fps  to start with.
Enjoy


Short Tutrial for Setting up Crop_Rec Module (3.5k 1:1 Preset) on the 5D Mark ii

For best results start with 30 fps ntsc in the canon Liveview menu
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgXcHYA%2FVRAM6-Small.png&hash=4e02fef66ed566c1d81f3422242de952) (https://imgbb.com/)

Load crop_rec  & mlv_rec , mlv_snd can be load but reduces the amount of frames recorded .
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjXKT6V%2FVRAM7-small.png&hash=04fec276e6b3ff8d1a3f918437e8b598) (https://imgbb.com/)

Next setup Liveview preview , This is the most important part without this set correctly it will not work !
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgKpLmV%2FVRAM8-small.png&hash=00ce5913c31ea07ac12dc74c869ddc17) (https://imgbb.com/)
Liveveiw needs to be set to ML Gray Scale
Set bit depth to 10bit & aspect ratio to 1:1

Next go in to the crop_rec menu select 3.5k preset .
Note ! this only work when you enter 5x zoom/3x crop_mode as per the hi-lighted texts says
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdTBuDA%2FVRAM9-small.png&hash=17908a3fa755be619e48da72590359fc) (https://imgbb.com/)

Once to go to 3x crop_mode you may need to go back to 1:1 then back to crop_mode  to get
2880x1080 sometimes , this is what you should see in the Movie tab once successful
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FmgmSYA%2FVRAM10-small.png&hash=5aa2b7a5c101157521f5ccb7c891437a) (https://imgbb.com/)
Note the Red hi-light box the Frame Rate now says 23.976
Now you see the "crop_rec" is enabled & the resolution is now 2880x1080

Now go to the Raw Movie Tab you should see the following
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fd2OGtA%2FVRAM11-small.png&hash=4602186922c01f068b82810dee2c31ae) (https://imgbb.com/)
You can choice resolutions less then 2880 if you wish , and you need to have ML Grey Scale selected

In Liveview this is what it should look like
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FnimQmV%2FVRAM13-small.png&hash=cbc528fd3a1a25c2587bd8e5335ec5bf) (https://imgbb.com/)
Notice the yellow circle "5x zoom"
Green hi-lighted area "Frame Rate" 23.976
The Red Hi-lighted area "crop_rec 1:1 3.5k preset "

For full color High Res preview press the half shutter
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FfUo20q%2FVRAM14-small.png&hash=de155b32660a9ad307e99dc2c538d151) (https://imgbb.com/)

Known Limitations
Once recording started you will have a Frozen B/W Liveview .
At this point it's unknown how to fix it yet , but Liveview return after recording stops.
In the current state you should see about 20 to 30 second of 10bit video.


New Update:

July 20 2020
Experimental build:Crop_Rec.Mo Updated for 4k3x1-3k1x1-2.8k1x1-48pfhd3x3
5D2-4k3x1-3k1x1-2.8k1x1-48pfhd3x3_7-10-2020_8.42pm_crop_rec.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5D2-4k1x3-3k1x1-2.8k1x1-48pfhd3x3_7-10-2020_8.42pm_crop_rec.zip)
Updated the Crop_Rec Module for multiplies of 3xcrop_mode/5xZoom presets .
Now there is 3k 1:1 (3016x1080 23.98fps) & Centered 3.5k 1:1 (2784x1160 24fps) in the same build plus 4k(3x1) & FHD 3x3 48p (1880x774)
With the new updated mlv_rec. module you get 8 more pixel in 3k & 24 more pixel in 48p 3x3 when use together .
Short Video tutorial on how to enable the new crop_rec presets in the latest update - (shot my 50D in 1x3 (1376x1742 @24fps ->4128x1742 from mlv. app)  ;)



July 5 2020
Experimental build:MLV_Rec.Mo Updated for 1880 (3x3) 2152(5xZoom)
1880_3x3-2152_5xZoom-5d2_7-5-2020-5.59pm-mlv_rec.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1880_3x3-2152_5xZoom-5d2_7-5-2020-5.59pm-mlv_rec.zip)

In Canon default  resolution 3x3 you gain 24 more pixels (1856->1880) & in 3xcrop_mode(5xZoom) there a gain of 8 pixels (2144->2152)
Will load on top of the Crop_Rec eXperimental builds , did not test with the nightly builds
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FJvnTghP%2FVRAM103-small.png&hash=bf14c7dfb4a01375ff1e5526ea3a7c2d) (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F0cWP49X%2FVRAM104-5x-Zoom-small.png&hash=fee840fcccb35d03da13727b82bd0fce) (https://imgbb.com/)

Jan 5 2020
Experimental build:5632x3124 @ 3.8fps
crop_rec-5632x3124_3.8fps-5D2-eXperimental.9.42pm.2020Jan05.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/crop_rec-5632x3124_3.8fps-5D2-eXperimental.9.42pm.2020Jan05.5D2212.zip)]
This a 1:1 pixel 5632x3124 @ 3.8fps , Very eXperimental you need at least a 1066x CFcard for this .
Read the discussion in these posts
https://www.magiclantern.fm/forum/index.php?topic=19336.msg223937#msg223937
https://www.magiclantern.fm/forum/index.php?topic=19336.msg224036#msg224036

Dec 31 2019
Experimental build: 4k(4096) 3x1 Anamorphic ,  2784x1160 1x1 @ 24.00fps & FHD 3x3 (1856x774) @ 48p
crop_rec-4k3x1-2.8k24p_3x3-48p-all-centered_8.18pm-5D2-eXperimental.2019Dec31.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/crop_rec-4k3x1-2.8k24p_3x3-48p-all-centered_8.18pm-5D2-eXperimental.2019Dec31.5D2212.zip)
In proved 4k-3x1 HDMI preview at 480p & changed frame rate to 24.00fps , reconfigured 2784x1160 for true 24.00 fps & realtime preview,
(1856x774  2:40 A.R. @ 48.028fps) write speed is 82MB/s so close to continuous
reduced the rolling shutter in 48p from 29.7ms to 17.8ms  :D

                                                                                                                     Short video on how to enable 48fps preset


                                                                                   Short test with 48fps in FHD (1856x688) then slowed down to 23.976fps & 12fps , shot @ 1/91th second , 400ISO lens @ f8

Same as before , it may lockup , freeze , etc. ... you guy know the routine
48p can be hard to enable , what I did was first put the cam in 1:1 (3x crop_mode)
enable 4k Anamorphic for a few second then switch to "FHD 48p" press the play but twice .
It will say in the left corner 1:1 but it's really 3x3 FHD , need to change that to say 3x3 .
FYI about 48p & shutter speed , to get correct shutter speed for 48p you should be double
your frames rate e.g. 24fps = 1/48th , 48fps = 1/96th of second ideally so the best I can get is 1/91th of second
to get that I set shutter speed in 3x3 without crop_rec enabled to 1/41th or 1/40th of a second then enable FHD 48p
should be 1/91th of a second , this give very good motion blur for slow-mo shots down to 12fps

August 21 2019
Experimental build: Dual ISO Video Module
dual_iso_video_5D2.mo (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dual_iso_video-updates_8-21-2019.zip)
Works with my crop_rec builds (not too sure about the nightly builds didn't test them) at 3x3 FHD(1856x1248) & 5xZoom(2144x1074) (3x crop_mode) @ 10bit raw ,
I didn't check it with the Crop_Rec Preset , so if you do try it with Crop_Rec it may freeze you cam and may need I battery pull .
Best to start out at 100 ISO with the recovery ISO at 800 , so 100/800 ISO , I wouldn't recommend a recovery ISO higher then 1600 ISO (200/1600 ISO Max.)
Anything beyond  that is just noise -- remember 5D2 has only 5 native analog ISO's (100 , 200 , 400 , 800 , 1600) the rest is just digital push or pulled ISO (e.g. 160 ISO = 100 ISO digital pushed)
The Zip file is just the dual_iso.mo (module) not the full build , if you think you need the full build with dual_iso.mo let me know , but it should work on top of
my builds on the first page , in theory it should work with the 5D2 10-12bit  magic lantern experiments


Oct. 22 2019
Experimental build: 50D ADTG_GUI Module Crop_Rec experiment 3744x1080 @ 23.976 fps
adtg-gui_crop_rec-3.7k-50D-eXperimental.2019Oct01.50D109.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/adtg-gui_crop_rec-3.7k-50D-eXperimental.2019Oct01.50D109.zip)
Hey 50D users I've started to port 50D to crop_rec , first step was to code a simple preset in ADTG_GUI.mo where a1ex first tested the 5d3 code.
Reasons why I put the 50D crop_rec in adtg_gui is to make sure it work or course .

How to get started:
Load mlv_rec.mo & adtg_gui.mo
Enable adtg_gui go to the advance menu and enable ENGIO Registers

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FVvfN2g6%2FVRAM0-small.png&hash=4589146cad995d7072044f016096af03) (https://imgbb.com/)

Press half shutter to exit to liveview then press the play button (make sure to have a cr2 there to view ,this refreshes liveview & enables the other 1500 reg's)
go back to the advance tab in the adtg_gui.mo and now enable the 3k 1:1 50D preset

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F48Wwqzk%2FVRAM2-small.png&hash=fb5a351edc6c965998e40b3ce0476cb7) (https://imgbb.com/)

Press half shutter to exit to liveview , then you may have to cycle thought the 5xzoom button ( but you should only have to press it once)
Make sure you set the shutter to as close to 1/48th of second as you can , I was able to set it to 1/49th and when the preset is enabled it will
go to 1/48th of a second  for 23.9876 fps @ 3744x1080

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F6XXtmBW%2FVRAM3-small.png&hash=b9b22f35dcd1f35fadcc335cc9995d36) (https://imgbb.com/)

Don't worry about the line on the right had side , there will not be in the raw image (this is the same thing that happen with the 5D2 , still working on a fix )
To make sure everything gets loaded , you can check the memory patches in the debug menu you should have 7 like this

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FxJFSvcv%2FVRAM4-small.png&hash=b2ffe9a65707113ae556557d9d412a6d) (https://imgbb.com/)

On a Lexar 1066x 64GB CF card was getting around 10 second in 10bit .

This is very Bleeding Edge Stuff , so expect lockups , cam liveview freezes , etc. .... if it brakes you get to keep both halves  :P



New Update: November 14/2019
3008x1080 @ 23.98 fps Updated Corrected Horizontal Centering
magiclantern-crop_rec-3k_Updated_Center_4.20pm-5D2-eXperimental.2019Nov14.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-crop_rec-3k_Updated_Center_4.20pm-5D2-eXperimental.2019Nov14.5D2212.zip)
Updated the cmos[2] Horizontal offset to correct centering offset ,  used full res CR2 (5613x3744) for referencing center as this post explained here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg222262#msg222262)

****************************************************************************************************************************************************

Ok by now a lot  if not all ml user know about crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView  (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182052#msg182052)
Yes as it says I'm looking in to 3K/UHD on the 5d2 cam and being that most of the digic iv cam are similar
My hope is it will help on other cams like 7D , 50D , etc.... in reality only cam with CF Card have the potential
of recording 3K/UHD other SD cams maybe in short busts or reduce frame rate but I think the 5d2 and 7D should be capably
of 3K or close to it . That's my target , right now 10bit 2144x1076 @ 23.976p is a reality at about 60-63 MB/s Write speed
with a crop factor of 2.61 . Some may ask if this is possible on cams other then the 5d3 , will the whole discovery came from a
500d experiment (http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909) from Greg , at least that's my understanding . To my knowledge a1ex reproduce the experiment on 5d3 digic v
which ended up code in the adtg_gui module (https://bitbucket.org/hudson/magic-lantern/commits/50d8f06018)  from iso_research branch . That's what I'm basing my development on and hints that a1ex gave my here (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182149#msg182149) ,
so the first step was to compile a adtg_gui module from the iso_research branch.
Of which I had many problem , thank to dfort for helping me with this issue the thread can read here (http://www.magiclantern.fm/forum/index.php?topic=19307.msg182371;topicseen#msg182371). Once I load the module I can now get in to the cmos H & V.
I loaded the 5d3 3K registry preset in 3x crop mode and start play round .
I what so far I have discovered with the vertical registry I'm able to move the crop window from the left most leading edge to the right leading edge in 46 increments.
Stating at cmos[2] @ 0x10c <--0x40E
Code: [Select]
0x1c
0x2c
0x3c
0x4c
0x5c
0x6c
0x7c
0x8c
0x9c
0xac
0xbc
0xcc
0xec
0xfc
0x10c
0x11c
0x12c
0x13c
0x14c
0x15c
0x16c
0x17c
0x18c
0x19c
0x1ac
0x1bc
0x1cc
0x1dc
0x1ec
0x1fc
0x20c
0x21c
0x22c
0x23c
0x24c
0x25c
0x26c
0x27c
0x28c
0x29c
0x2ac
0x2bc
0x2cc
0x2dc
0x2ec

When I have any new developments  or share any info I'll post here and anyone that's wishes to help or build on my work Please join in
looking at you @Ilia3101  ::)   ;D



Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on April 06, 2017, 12:45:30 AM
I'll be standing by you with all of my support for this. Also own a 7D and a coworker of mine owns a 5D2 whom is more than willingly to loan to me for to test along. Thanks for doing this @reddeercity!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 06, 2017, 03:25:04 AM
Juicy!
I will do whatever is useful. If you need me to try hundreds of values on registers I can do that. Will be back in a couple days and ready to do stuff(excited to look at the source code :o and the Gr3g resolution hack)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 06, 2017, 05:42:57 AM
@ DeafEyeJedi good to hear.

@ Ilia3101 sound good , if you go to my bitbucket_downloads (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/) there's a atdg_gui build for 5d2 that I'm using ,
get that and load the adtg_gui , raw_rec & file_manger modules keep it simple . In the submenu of adtg_gui in Advance , toggle on the digic registers
then at the bottom of the menu you will see 1:1 3K crop mode  (5d3) load that preset and go back to the root of adtg_gui and you will see 2 Green icon by cmos[1] Vertical  cmos[2]Horizontal
beware you will get lockups & liveview freeze , just pull the battery and restart etc.... . I use frame over ride @ 23.976p . Start reading the code here (https://bitbucket.org/hudson/magic-lantern/src/50d8f06018dc52f72f04265432dd83106e495e85/modules/adtg_gui/adtg_gui.c?fileviewer=file-view-default#adtg_gui.c-144) for hints & info on the registers plus look at this  post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182149#msg182149) for more help.
From what I understand it's a combination of registers & video timers according to a1ex.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 06, 2017, 07:02:11 AM
Thank you. I've played around with the atgdgooi branch before, to be honest it scares me a little, as I suspect I've done damage to the sensor (I think it increased the fixed Patten noise) by playing with it. I also discovered 'green_rec' along with 'magenta_rec' and 'vertical_bar_rec'.
Will try out the 3k presets when I get back. The goal is to find the right values of those registers for the other cameras(5D2) to increase resolution in the same way? Or would it be controlled by other registers on other cameras? I guess this what we need to find to get it working. I will be reading the wiki for some homework ;D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 06, 2017, 08:35:46 AM
The goal is to find the right values of those registers for the other cameras(5D2) to increase resolution in the same way?
more or less , but first 5d2 then others .

Something interesting
cmos[1]0xaac <--0xeac
cmos[2]0x404 <--0x10E
looks like almost full view compared to the frame on the right which was not modified
the left all squeezed together .  I was using a EF24-70f2.8L at 24mm so in 3x crop mode that's 24mmx2.61=62.64mm
so I would say the left frame is very close to 24mm

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2858%2F33055712593_8481a985f8_b.jpg&hash=48b34f65cbcee570e5f038cc46b17d71) (https://flic.kr/p/Sn2aHc)
full view crop mode (https://flic.kr/p/Sn2aHc) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Exported Jpeg from MLVProducer , I pushed the exposure to show detail.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2852%2F33055724173_5696406a53_b.jpg&hash=ce2800c02f2c1aac9e64cdb954031342) (https://flic.kr/p/Sn2e9R)
M05-2224_00001 (https://flic.kr/p/Sn2e9R) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Non modified registers cmos cdng 2K
https://www.dropbox.com/s/hzds2gbijp4tlzv/M05-2336_00000.dng?dl=0
Modified cmos[1] & cmos[2] cdng
https://www.dropbox.com/s/ng1zbz3xy8auopp/M05-2224_00000.dng?dl=0

It's a start  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Deadcode on April 06, 2017, 08:49:58 AM
Cheers! I hope you make it!

Is it possible to implement the lossless compression with older Digic 4 cameras?
With my calculations, the highest continuous resolution in crop mode might be 2736x1164 at 23,976 fps, with 2,1x crop ratio. That requires 128MB/s write speed and with the "usual" 58% compression is just about 74MB/s. That's around the max stable write speed with the 5D2 in crop mode.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 06, 2017, 08:59:32 AM
@Deadcode The compression is definitely possible in the 5D2 as it also has those features in the firmware that are being utilised on the 5D3 right now, but I think it takes an a1ex level of digic programming skills to do. So possibly may happen... or not. You can never predict.
@reddeercity squished crop mode would be pretty good, looks like there's a big black area on the right of that frame though. I bet getting rid of that pink corruption will be the hardest part.
Great start though!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 07, 2017, 02:44:24 AM
Yea I think compressed raw will work , I did a test merge of the compressed_raw branch in to unified source code and have a little problem with src/raw.c merge  that I hadn't had time to look in too fully but no crashes on the build .

I bet getting rid of that pink corruption will be the hardest part.

No , shouldn't be too big of a problem , I can clean up the corruption by changing addresses . Main thing I'm looking for is the full View with the right aspect ratio right now .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: goldenchild9to5 on April 07, 2017, 08:05:44 AM
@reddeercity Great work..
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on April 07, 2017, 08:11:05 AM
Some of the values are actually bit fields. My old notes on 5D2 registers:

Code: [Select]
CMOS[0]: 0x203 at ISO 100
---- ------xx default 11, setting to 00 results in dark image with very low stdev
---- ---xxx-- ISO, values from 000 to 100, ascending (ISO 100-1600)
---- xxx----- second ISO field (as in dual ISO), disabled by default (see next bit)
---x -------- enables second ISO field (default 0)
--x- -------- enables vertical OB clamping maybe (setting it to 0 results in severe horizontal banding that looks like random walk)
-x-- -------- compresses the image horizontally (left side squashed, right side black)

CMOS[4]: 0x242 at iso 100...800 and 0x244 at iso 1600
---x -------- looks like some vertical dual ISO
-x-- -------- seems to cleanup 0.2 or 0.25 stops of shadow noise
No effect observed on the other bits.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on April 07, 2017, 01:45:18 PM
@reddeercity Can't wait, but what is the atdg_gui build file for? So far the only way to use 10bit or your possible build is to push the magnify but for 5X to put in crop mode and you have to have a lens like 24-70mm 2.8L lens to view?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 07, 2017, 11:09:29 PM
@reddeercity Can't wait, but what is the atdg_gui build file for?
For developing
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 07, 2017, 11:19:31 PM
Some of the values are actually bit fields. My old notes on 5D2 registers:

Code: [Select]
CMOS[0]: 0x203 at ISO 100
-x-- -------- compresses the image horizontally (left side squashed, right side black)
Thanks  :) , interesting that's on cmos[2] 0x404 reproduces this .
I found more on Greg's work last night read thought the CMOS/ADTG/Digic register investigation on ISO thread , and it seems
I may have been adjusting the wrong registers (cmos 1&2) it seems he was adjusting cmos 5 &6 for digic iv cams
I'll give that a try .
No ISO related , seems it has more in common to 5D3
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 08, 2017, 09:51:39 AM
Some more update/process , Getting cleaner image now with some surprise's  :D

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F4%2F3952%2F33776580001_4c3b4d137a_b.jpg&hash=1d9d2d8a71e80df5746ac9cd09af94f7) (https://flic.kr/p/TsHNtB)
M07-2145_00001 (https://flic.kr/p/TsHNtB) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr   5643x1076


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2819%2F33776587241_a225ae18b8_b.jpg&hash=02d0170c9ca6bdf91460d0b8d1e0818c) (https://flic.kr/p/TsHQCr)
M07-2149_00157 (https://flic.kr/p/TsHQCr) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr    5643x1076

Ok I had to do a little post work to get it to the right Aspect Ratio   ::), I still getting the compressed image but it's seem to be all there.
I crop off the black area then just increased the horizontal by 300% and left the vertical unchanged

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F4%2F3727%2F33062768594_ef793a51fb_z.jpg&hash=3fd4e3400aa94f361eba44ba14204aaa) (https://flic.kr/p/SnDkdo)
M07-2145_00001 (https://flic.kr/p/SnDkdo) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F4%2F3816%2F33062774114_e01881b6ae_z.jpg&hash=5e5a8675f517a7df424ec3f587acd647) (https://flic.kr/p/SnDmRy)
M07-2149_00157 (https://flic.kr/p/SnDmRy) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

Screenshots from crop mode , showing the capture area that's what recorded for reference only

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2930%2F33062795444_28a1527301_b.jpg&hash=cf3c436ba332605148ee23731a4c9e3b) (https://flic.kr/p/SnDtcj)
5D2 Liveview (https://flic.kr/p/SnDtcj) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

I'm having problem with the FPS timer A , it doesn't want to go below 610 , I have no problem with timer B in fact I had it up to 3000 that help clean up the frame
it seems timer A is lock and I think that what stopping me for getting the full width capture not compressed to 2K frame , I had at about 17 fps to get a clean image

Or do I have to play with the Head Timers? I know there's three Can't find any info on them , I believes I need to get timer A to 455 or in that range and timer B about 2200 .

Cheers
Edit: This is frame with FHD 1856x1044 I used on my registers crop full view test , for reference only
 (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2889%2F33777229671_62d61665f6_z.jpg&hash=abe96c65f087712e2523481b1062e89f) (https://flic.kr/p/TsM8AP)
M07-2337_00001 (https://flic.kr/p/TsM8AP) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 08, 2017, 11:09:44 AM
Good work. Does the black area go away if you move the crop window by the right amount? Yay now we have the 5D3 3x1 binning from crop rec!
Edit: technically the black bar can't go away unless you set the resolution to < 1881. Also that's insane spectrum ratio at 5.25:1
@a1ex genius. Are the registers 12 bit? That's weird ::)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 09, 2017, 07:28:34 AM
I'm getting more stability now ,
the registers I'm using is:
ADTG1 [105f]N-----0xa41
CMOS[1]------------0x2c7
CMOS[2]------------0x404

Exact FPS
FPS timer A --------610 (can't  lower it for some reason)
FPS timer B -------2000
main clock---------24.00 Mhz
Actual FPS---------19.672

Image is very clean now , no artifacts.
but image is still compressed to the left , I believe that because of FPS timer A is not be able to lowered from 610
I think there's something I'm missing , I feel I'm getting very close now If I can figure out the video timer problem .

Edit:

ADTG1 [105f]N-----0xab56
CMOS[1]------------0x2c7
CMOS[2]------------0x404
CMOS[5]------------0xc05

New setting that open up all registers , before I couldn't figure out way I didn't see the registers at where documented in other posts.
By accident I started to play around with CMOS[5] and BOOM there is all these Registers  :o
there go on forever . Now I access more RAW H&V and video timer stuff.
 

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 10, 2017, 04:48:41 PM
@reddeercity I've now tried the iso-research branch and setting registers, your settings didn't work too well for me, I managed to get the squish working with cmos[0], but black level kept going up and down causing it to all look ugly, setting cmos[1] and others to values you recommend didn't help much other than locking up the camera. Basically i've not had any success.
But one more thing: when the correct registers/settings to unlock resolution are found, will it automatically be detected in the raw_rec or mlv module so the higher resolution can be set straight away?(I mean: does it auto detect max resolution?)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: nikfreak on April 10, 2017, 04:57:54 PM
once the sweet spot(s) for 5D2 are found you could for e.g. adjust/activate your new "crop mode / resolution" via ML menu for the moment. Check a1ex's 2016 implementation (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182149#msg182149)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 11, 2017, 02:29:16 AM
@Ilia3101 ,when selecting registers I find it's more temperamental the warmer the sensor gets . There is a certain  order you have to do and of course the video timers have to adjusted or it will not work not matter what you do. 1st set to low light , video timer A ( low as it can go) , video timer B (2000 and up) works in the adtg_gui  advance menu enable digic registers then apply the 1:1 3k preset (which sets cmos[1] & cmos[2] the I adjust cmos[2] first for width then cmos[1] for height next is ADTG1[105f]N (x5 & x10 zoom shutter blanking), move the frame up & down by either by shorting or lengthen shutter duration, to fix pink frame you may have to find a different registers in cmos[1] oh yea also enable cmos[5] helps with pink hi-lights you may also need to play with frame over ride a bit , seems that the frame is clean around 18-20 fps not sure why. Also It looks like I may have got the 1:1 Crop_record working on the 5D2 so you don't have to be in zoom mode for it to work  ;D  As it say 1:1  but at 1856 not 5616 but I can move the capture area cross the sensor just like 3x crop ,  plus in that mode I was able to record a H264 in 1:1when I turn off raw video but the problem I having and you know there's always a problem  ::) I can only record 1856 width(center crop) but full height which is compressed vertically , once I stretch it out it's 1865x3744 .

1:1 Crop_recording with 3K preset , This is Not 3x Crop mode but 1:1 , like I said with compressed vertically . 1856x1044

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2943%2F33579296740_8b26e7bcfd_z.jpg&hash=ead75692246fe718ec5092e860fdc833) (https://flic.kr/p/TahF35)
M10-021100001 (https://flic.kr/p/TahF35) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr 

After I resized the vertical by about 258% I get 1856x2694 close to full height of the sensor but at 16x9
 
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2908%2F33151750183_9c03de8ab8_z.jpg&hash=05e88ae33de71e30cce28b270fb5eb65) (https://flic.kr/p/SvvokH)
1856x2694-M10-021100057 (https://flic.kr/p/SvvokH) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

1:1 crop_record H264 from the 5D2 , turn off the raw video and hit record  :) , Maybe Highly Compressed h264 3K or 4K  :P :P
Just click on the image and ficker will link to the file on page. or watch off dropbox see link below

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F4%2F3871%2F33579323710_ef0f247671.jpg&hash=3bac3fa5b97fa79a53f159d45df00681) (https://flic.kr/p/TahP45)
MVI_8653 (https://flic.kr/p/TahP45) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

So how do I know if it's truly 1:1 , will I don't really only thing that tells me is the fact there no moiré or aliasing pattern
look at the the fine hole on my MacPro even in h264 I can't tell , thou the lens I use was 24mm so no lens correction
which give the distortion width at the top .

H264 1:1 crop_record MVI_8653.MOV (https://www.dropbox.com/s/06mqxxa40aqkupf/MVI_8653.MOV?dl=0) source file link from my dropbox if any one what to download 45Mb

 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 11, 2017, 09:01:21 AM
Doing some more quick tests in 1:1 LiveView with the following setting
adtg1[1061] 0x449
cmos[1] 0x3e0
cmos[2] 0x104
cmos[5] 0xc21

23.976 from 24
Video Timer A 584 (FT +12)
Video Timer B 3422 (FT +1672)
Actual frame rate 12.00

crop_record 1:1 (Not 100% sure it working but seems to setup a lot easier and centered
1:1 3K preset that's it .

3000x1250
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2872%2F33839219381_57645a82f5_c.jpg&hash=86d1dec5205393d9aae051143c2c4cd7) (https://flic.kr/p/TyfQYV)
M10_3000x1250 (https://flic.kr/p/TyfQYV) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


3000x1250
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F4%2F3933%2F33928144306_f1aabc196d_c.jpg&hash=6a1f5bf7a1f08646b475650ecd650835) (https://flic.kr/p/TG7Bgh)
M10_2_3000x1250 (https://flic.kr/p/TG7Bgh) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

There's a little trick I did to get this to work , I post tomorrow on what the special thing is  8)
The hint is in the frame size   ;) 

ProRes files of the 3K test  , not big  about 100Mb each
M10_1.mov (https://www.dropbox.com/s/0475byz3iw23csi/M10_1.mov?dl=0)
M10_2.mov (https://www.dropbox.com/s/1onvbkm14l0indn/M10_2.mov?dl=0)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 11, 2017, 09:27:40 AM
I see the hint: 1250 ;) Maybe you got something like the vertical compression in the post before but sideways? just a guess...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: goldenchild9to5 on April 12, 2017, 01:00:39 AM
@reddeercity Great job  :) Now I wanna buy a 5D2 as a back-up to my 5D3. 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 12, 2017, 02:51:07 AM
This will not work on old cameras (500D, 5D2 etc) until you find a way to change the EDMAC size.

Old cameras do not support #ifdef CONFIG_EDMAC_RAW_SLURP
https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-429

Maybe patch here :
Code: [Select]
LiveViewMg:000929e0:00:00: *** ConnectWriteEDmac(0x10, 0x0), from ff1f7134
LiveViewMg:000929e0:00:00: *** SetEDmac(0x10, 0x463390a4, 0xff44e5c8, 0x20000000), from ff1f7164

Or try "#define CONFIG_ALLOCATE_RAW_LV_BUFFER"
https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-126
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 12, 2017, 03:05:24 AM
Thanks for the info Greg I looking for something like this
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 12, 2017, 04:22:01 AM
So would EDMAC#5 cofo4500
address:43e28a4
State : 1
Flags : 20000040
Size : B (4046 x1127
Connection : write=0x0    read=0xff

found that in "Show EDMAC" in the debug menu
Look like the right frame size if that's what the 4046x1127 is ?

Or I'm I miss understanding this .

Looking at this #crop_rec_4k-raw.c-479 (https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-479) , so if I replace "EDMAC+8" with "EDMAC+5 ,  will that work or I'm I off base here ?
Code: [Select]
   /* autodetect raw size from EDMAC */
    uint32_t lv_raw_height = shamem_read(RAW_LV_EDMAC+4);
    uint32_t lv_raw_size = shamem_read(RAW_LV_EDMAC+8);
    if (!lv_raw_size) return 0;

    int pitch = lv_raw_size & 0xFFFF;
    *width = pitch * 8 / 14;
   
    /* 5D2 uses lv_raw_size >> 16, 5D3 uses lv_raw_height, so this hopefully covers both cases */
    *height = MAX((lv_raw_height & 0xFFFF) + 1, ((lv_raw_size >> 16) & 0xFFFF) + 1);
    return 1;
#endif

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 12, 2017, 06:40:09 PM
4046/14*8 = 2312
So the size is 2312 x 1127

500D "LV_Initialize %s" :
Code: [Select]
int a1 = 0x1D78 + 0x2600;
*(a1 + 208) = 1664;
*(a1 + 210) = 660;
It looks like width and height raw, but who knows.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 13, 2017, 08:47:54 AM
That's Greg  :)
Now the Crazy Part  :o

Some how either with dumb luck or divine intervention ::) I manage to get 3.5K in 1:1  @1.6 crop  8) 8) 8)

Check it out , thou I couldn't record any image just yet had a small error in edmac-memcpy.c line 80

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F4%2F3944%2F33622003720_5fcb760cb6_o.jpg&hash=02bbb8f608c1e68cf80802621dfd4965) (https://flic.kr/p/Te4ykj)
5D Mark ii experimental 3.5K Raw Video (https://flic.kr/p/Te4ykj) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

The write requirements for that size was reported about 198MB/s @30fps 14bit :o
If and when I get if working fully then compressed raw is next.

The files I modified are raw.c , edmac-memcpy.c
in raw.c I add this
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif

also in raw.c  changed to 0xC0F04500 from  0xC0F04508
Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04500
#endif


in edmac-memcpy.c change the r/w channel to 0xff & 0x0 from read 0x19 & write 0x03
If may be a problem as I have a write issue , need to look further in to this of course.

Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
uint32_t edmac_read_chan = 0xff;
uint32_t edmac_write_chan = 0x0;

That's a good step in the right direction and the start many buggy problem to come
But Hey I'm so much closer now thanks to all  :D
Poor mans 1DC  :P :P :P
more to come

Edit:FYI to get this build I clone the source ,  updated with iso-research branch then modified the files 
Title: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on April 13, 2017, 09:20:07 AM
Really incredible stuff and I am loving this progress so far, David!

'Poor man's 1DC' -- very well said!

Also after some thoughts in my head recently I remember helping @nikfreak out w 70D/100D merge awhile ago by doing this...


Guess it's basically like a cheat sheet by clicking on 'don't click me' then afterwards you go into EDMAC which will show a screen full of registers addresses after while in each mode (mv1080, mv720, 5x zoom crop, etc) and it seemed to help him out.

Maybe you already are aware of this protocol but felt it was worth sharing just in case.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: nikfreak on April 13, 2017, 09:49:03 AM
I can recommend to undo all code snippets from above. Yor best bet is to try to get near 3k width (http://www.magiclantern.fm/forum/index.php?topic=19336.msg182706#msg182706) with adtg / fps overrides
Don't change the EDMAC channels. They have proven to work for years and there's find_free_edmac_channels() if you really need to add more.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 13, 2017, 10:58:32 AM
@reddeercity Wow, difficult to believe thats happening :o :o :o I was a little pessimistic about getting rid of that pink corruption, but you got rid of it, so I guess you'll be able to fix the record error ;D I can't be much help developing this stuff, as I don't understand how the the Magic Lantern code even works and I'm not a lot of use at programming in general. I'm always ready to test tho :D :D

@nikfreak What is the level of risk in doing this kind of stuff, corrupted image or corrupted ROM?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: nikfreak on April 13, 2017, 11:19:01 AM
Low risk of a brick if you know what you do. You got to keep all pieces if it breaks ofc.
You can get raw recording errors, corrupted frames or wrong recording sizes etc.

http://www.magiclantern.fm/forum/index.php?topic=19336.msg182847#msg182847

follow the links to Greg's PoC and a1ex's 3k implementation.(Digic5) They did it just with modifying register(s). The difference between Digic5 and Digic4 is:

Ofc you must find apropriate value for 3k width for 5D2. Don't think that 5D3's will match 5D2.

Check this code snippet (https://bitbucket.org/hudson/magic-lantern/src/50d8f06018dc52f72f04265432dd83106e495e85/modules/adtg_gui/adtg_gui.c?fileviewer=file-view-default#adtg_gui.c-126)

I guess Canon did some calculation themself befor implementing zoom/crop modes for FHD H264 30FPS recording. Now you can calculate yourself how much you can extend the resolution in zoom/crop mode (x*y*fps) for recording with 24fps. Maybe 15-20% could be fine but you might have to extend the buffer, too...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 14, 2017, 08:59:02 PM
500D  LV zoom 5x - "StartPass_x5 CrawAddr:%lx / Kind:%d"
Code: [Select]
SetEDmac(0x10, *(0x16604 + 808), 0xFF44F0B0, 0x20000000);

ROM:FF44F0B0
ROM:FF44F0C8        DCD 0xE1C, 0      //‭3612‬
ROM:FF44F0D0        DCD 0x35D, 0, 0   //861

5D2  LV zoom 5x - "StartImagePass_x5 CrawAddr : %lx / KindOfCraw : %d"
Code: [Select]
SetEDmac(0x5, *(0x13ECC + 804), 0xFFCAA1D4, 0x20000000);

ROM:FFCAA1D4
ROM:FFCAA1EC        DCD 0xFCE, 0       //‭4046
ROM:FFCAA1F4        DCD 0x467, 0, 0  ‭  //1127‬


This code will increase EDMAC size, but the buffer will be overflowed - it's dangerous.
Code: [Select]
static void run_test()
{
    static uint32_t craw_5x_edmac_h = 0xFF44F0D0;
    patch_memory(craw_5x_edmac_h, 861, 861 + YRES_DELTA, "lv_edmac_h"); // add height
}

Now we should use our own buffer.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 15, 2017, 04:03:14 AM
Thanks for the code but I'm not really sure how implement it  , being read the source code raw.c & edmac_mencpy.c
for clues still not sure , can you point me in the right direction please.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on April 15, 2017, 08:08:14 AM
I'm keen to get this going on the 550D.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 16, 2017, 02:27:06 PM
I increased the height by 140px (841px to 981px). More does not work.
I do not know why, some problem with memory or edmac.

To do this you need to change (500D):
Code: [Select]
CMOS[5] //vertical offset
ADTG[2]1172 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1173 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1178 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1179 -> old_val + YRES_DELTA (nrzi)
ADTG[2]1214 -> old_val + YRES_DELTA (nrzi)
ADTG[2]120A -> old_val + YRES_DELTA (nrzi)
C0F0713C -> 0x370 + YRES_DELTA
C0F07150 -> 0x388 + YRES_DELTA
C0F06088 //RAW last line|column
C0F06008 -> 0xaf90af9   //photo mode value
C0F0600C -> 0xaf90af9  //photo mode value
C0F06010 -> 0xaf9    //photo mode value
C0F06014 -> 0xc83   //photo mode value
patch EDMAC size
patch buffer
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 16, 2017, 09:33:28 PM
Buffer patched :

Code: [Select]
void* buf = malloc(2064 * 2000 / 8 * 14);
MEM(0x16604 + 808) = buf;

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs16.postimg.org%2Fy4mp9zyph%2F500d-lv.jpg&hash=4626a52d5dc0951d4fc9b860589bc7dd)

It still works a bit randomly, some frames have standard height...

So if someone finds a solution, it will work.
I do not plan further research.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 17, 2017, 01:29:51 AM
@Greg Cool thanks , I'll keep going .
Just a couple of questions.
Was this test with the 3x Live view code & edmac size buffer patch   or just buffer patch ?

I still haven't tried the code you posted , need to find where I should add this code too .



Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 18, 2017, 10:59:26 PM
It was in live view 5x zoom. Both buffer patch and edmac size.

The buffer should not be needed :

silent.c
Code: [Select]
/* Reprogram the raw EDMAC to output the data in our buffer (ptr) */
raw_lv_redirect_edmac(sp_frames[next_slot % sp_buffer_count]);
sp_num_frames++;

raw.c
Code: [Select]
/* to be called from vsync hooks */
void FAST raw_lv_redirect_edmac(void* ptr)
{
    #ifdef CONFIG_EDMAC_RAW_SLURP
    redirected_raw_buffer = (void*) CACHEABLE(ptr);
    #else
    MEM(RAW_LV_EDMAC) = (intptr_t) CACHEABLE(ptr);
    #endif
}

So ML uses its own buffer, which should have the correct size after patching EDMAC.

If you do not solve this problem, we will announce the end of digic IV support. :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on April 18, 2017, 11:55:34 PM
The custom buffer is required, as ML redirects it only while recording.

In the crop_rec_4k branch I've used a SRM buffer (which can accommodate a full-res 14-bit picture). This needs to be passed to SetEDmac as the first argument (instead of Canon's default buffer).

The buffer redirection without CONFIG_EDMAC_RAW_SLURP is fragile: it relies on lucky timing. It's best refactored somehow, but cache patching is also ugly...

It's quite difficult to get these patches right, but I'm making progress on a tool that should help understanding the LiveView internals a lot better.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 19, 2017, 07:24:39 AM
@Greg thanks for the info
If you do not solve this problem, we will announce the end of digic IV support. :D

Yea  I had thoughts , but if there a will there way  :D

The custom buffer is required, as ML redirects it only while recording.
So when I use the custom RAW_LV_BUFFER_ALLOC_SIZE (2040*1267) , RAW_LV_EDMAC  0xC0F04508 & r/w channel to 0xff & 0x0 and got 3584x1068 (got all this from the "show edmac"  in the debug menu in 1:1)
(I forgot to mention in  Reply #27 where I got 3.5k , I enabled adtg_gui with digic registers and applied the 3k 5d3 preset)
So this happen by chance to have access to that lv buffer size ? Oh , I also had the Raw video frame size at the 3584 preset , thou I could not adjust the size or aspect ratio
it was lock to that size . Just trying to understand better .

Edit: A thought just came to mind , so if there was a higher preset  , lets say 4096 etc. .... instead of 3584 would it go to the highest possible size ? or is this limited to lv buffer size for that  frame size and I had frame override turned off with canon menu frame rate set to 30 ntsc

....... but I'm doing progress on a tool that should help understanding the LiveView internals a lot better.
Yes  :) That great to hear @a1ex
There's still hope for digic iv
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on April 19, 2017, 07:48:04 AM
Just FYI, I had an attempt (https://bitbucket.org/hudson/magic-lantern/commits/1ee967909f18d06f8af21d6f244fd1062fb59cfd) to find out the size of Canon's LiveView raw buffer on 5D3.

Result (not committed): it's not allocated using "traditional" malloc, but its address (and probably its size too) are hardcoded, near 0xFF16B318 (1.2.3). The size reported by the memory "allocation" routine was 0x1cae000 (30MB - enough for 10-bit full-res LiveView), but when trying to use that, I've noticed the bottom of the frame being overwritten.

So, I don't know the size of the default LV RAW buffer. On 1100D (a camera with only 128MB RAM), Canon's default buffer is overwritten (by other Canon code) even at small LiveView resolutions, so the safest way is to assume nothing about it...

Most models appears to handle x5 zoom fine though, but that's not a guarantee other tasks are not using this memory. Might even be the cause for some kind of frame corruptions...

(memory management on these cameras is quite strange, with many hardcoded (http://www.magiclantern.fm/forum/index.php?topic=5071.0) areas, fragmentation, and descriptions found in strings not always matching the reality)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 20, 2017, 09:05:27 AM
Thanks help a lot to understand , since I had a little luck with that custom lv buffer I used in post#27 I thought I would try and investigate further and see if any variable change EDMAC#5 in 1:1 & 3x crop . So I notice that in photo live view there is no secondly lv buffer for 5x zoom(3x crop) (could be that it's in 1:1 pixel mode so not needed I guess) but in movie mode there is and by switching Raw photo mode in the canon menu from raw to small raw1 & small raw2 the address changes as follows:
Code: [Select]
EDMAC Address 103e28a4  (3578x1267 1:1)(4046x1127 3xcrop) raw(5616x3744) (Movie mode)
EDMAC Address 103e28a4  (3578x1267 1:1)(4046x1127 3xcrop) small raw1(3861x2574)(Movie mode)
EDMAC Address 43e28a4   (3578x1267 1:1)(4046x1127 3xcrop) small raw2(2784x1856)(Movie mode)

EDMAC Address 103e28a4  (3578x1267 1:1)(No buffer change) (5x zoom) (3xcrop) raw (5616x3744) (Photo mode)
EDMAC Address 103e28a4  (3578x1267 1:1)(No buffer change) (5x zoom) (3xcrop) small raw1(3861x2574)(Photo mode)
EDMAC Address 43e28a4   (3578x1267 1:1)(No buffer change) (5x zoom) (3xcrop) small raw2(2784x1856)(Photo mode)

Not sure if this help , look interesting -- from this I'm concluding that 3X crop has a larger lv buffer maybe because in 1:1 pixel instead of line skipping and pixel binding
for FHD , just guess there. Next I think I'll try that large buffer and see if I can hard code the edmac address unless ML check for available addresses (no sure) , specially with the small raw2 and check the results .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on April 20, 2017, 09:19:00 AM
Uncompressed 14-bit data takes 14 bits regardless of what pixel binning mode is used. Should be obvious...

The addresses are allocated dynamically, so of course, changing the camera settings (or sometimes, just starting in a different mode) will give different numbers.

Photo quality shouldn't be relevant (it might change slighly the amount of available RAM, or how it's divided between SRM and shoot buffers), but you will have to use some memory allocation API to get a memory buffer.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 21, 2017, 04:50:17 AM
Ok ,understood .
..... you will have to use some memory allocation API to get a memory buffer.
Maybe this will help ? did a search for "memory allocation API "and got this , go figure developer ARMS api , could be a wealth of information , assuming I can understand it  ::)
https://developer.arm.com/docs/0030/latest/modules/memory-allocation

Edit: something interesting to look at schematic diagrams for 5d2 I think , takes me back to my early auto mechanic days when computers started (mid '80's ) to control basic engine functions
 which of course was very buggy  :P
https://photo-parts.com.ua/parts/Datasheets/KCD-A210-BACH8SchematicDiagrams_ENG.pdf
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 21, 2017, 07:00:17 AM
Some interesting info from the datasheet
Quote
32-bit input capture register : 4 channels
https://photo-parts.com.ua/parts/Datasheets/TMP19A43FDXBGTMP19A43FDXBG.pdf

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 21, 2017, 08:23:05 AM
I think I found register memory map  :D
http://www.ti.com/lit/ds/symlink/tms320dm368.pdf
page 50 of 208
Table 3-1. System Module Register Memory Map

Code: [Select]
Table 3-1. System Module Register Memory Map
Hex address Register           Acronym            DESCRIPTION(1)
0x01C4 0000                    PINMUX0            Pin Mux 0 (Video In) Pin Mux Register
0x01C4 0004                    PINMUX1            Pin Mux 1 (Video Out) Pin Mux Register
0x01C4 0008                    PINMUX2            Pin Mux 2 (AEMIF) Pin Mux Register
0x01C4 000C                    PINMUX3            Pin Mux 3 (GIO/Misc) Pin Mux Register
0x01C4 0010                    PINMUX4            Pin Mux 4 (Misc) Pin Mux Register
0x01C4 0014                    BOOTCFG           Boot Configuration
0x01C4 0018                   ARM_INTMUX      Multiplexing Control for Interrupts
0x01C4 001C                  EDMA_EVTMUX     Multiplexing Control for EDMA Events
0x01C4 0020                  DDR_SLEW           DDR Slew Rate
0x01C4 0024                  UHPICTL               UHPI Control
0x01C4 0028                  DEVICE_ID           Device ID
0x01C4 002C                  VDAC_CONFIG      Video DAC Configuration
0x01C4 0030                  TIMER64_CTL       Timer64 Input Control
0x01C4 0034                  USB_PHY_CTL      USB PHY Control
0x01C4 0038                  MISC                   Miscellaneous Control
0x01C4 003C                  MSTPRI0             Master Priorities Register 0
0x01C4 0040                  MSTPRI1             Master Priorities Register 1
0x01C4 0044                  VPSS_CLK_CTL    VPSS Clock Mux Control
0x01C4 0048                 PERI_CLKCTL        Peripheral Clock Control
0x01C4 004C                 DEEPSLEEP           DEEPSLEEP Control
0x01C4 0050 -                                          Reserved
0x01C4 0054                 DEBOUNCE0         Debounce for GIO0 Input
0x01C4 0058                 DEBOUNCE1         Debounce for GIO1 Input
0x01C4 005C                 DEBOUNCE2         Debounce for GIO2 Input
0x01C4 0060                 DEBOUNCE3         Debounce for GIO3 Input
0x01C4 0064                 DEBOUNCE4         Debounce for GIO4 Input
0x01C4 0068                 DEBOUNCE5         Debounce for GIO5 Input
0x01C4 006C                DEBOUNCE6          Debounce for GIO6 Input
0x01C4 0070                DEBOUNCE7          Debounce for GIO7 Input
0x01C4 0074                VTPIOCR               VTP IO Control
0x01C4 0078                PUPDCTL0 IO         cell pullup/down on/off control #0
0x01C4 007C                PUPDCTL1 IO         cell pullup/down on/off control #1
0x01C4 0080                HDVICPBT              HDVICP Boot Register
0x01C4 0084                PLL1_CONFIG         PLL1 Configuration Register
0x01C4 0088                PLL2_CONFIG         PLL2 Configuration Register

http://www.ti.com/lit/ug/sprufg5a/sprufg5a.pdf
More Registers (4.1 Memory Map) Page 25 Table 6. Memory Map & Page 27 Table 7. ARM Configuration Bus Access to Peripherals
http://www.ti.com/lit/ds/symlink/tms320dm368.pdf
page 85 Table 6-9. PRTC Interface (PRTCIF) Registers
page 86 Table 6-10. Power Management and Real Time Clock Subsystem (PRTCSS) Registers
http://www.ti.com/lit/ds/symlink/tms320dm368.pdf
page 91 Table 6-16. EDMA Channel Synchronization Events
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on April 21, 2017, 08:32:44 AM
Some interesting info from the datasheet https://photo-parts.com.ua/parts/Datasheets/TMP19A43FDXBGTMP19A43FDXBG.pdf

That's the MPU (http://www.magiclantern.fm/forum/index.php?topic=17596.0).

http://www.ti.com/lit/ds/symlink/tms320dm368.pdf

That's probably a chip similar to the one running JPCORE (http://www.magiclantern.fm/forum/index.php?topic=18443.msg177082#msg177082), but we could not confirm this hypothesis (http://www.magiclantern.fm/forum/index.php?topic=18443.msg177264#msg177264).

For memory allocation API (https://en.wikipedia.org/wiki/Application_programming_interface), see http://www.magiclantern.fm/forum/index.php?topic=8358.0 (and http://www.magiclantern.fm/forum/index.php?topic=5071.0 for some more low-level info).
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 23, 2017, 08:42:33 AM
going thought the branches I came across "new-lv-buffer-detections"  :o
as this is my issue atm , so I complied it are run it enabled the mem_spy.cfg & mem_chk along with raw video
not too sure how to use it correctly . mem_spy was running in the background but I was hoping it would make a log file but did not.
I was able to run the stub test in the "burn in test" with mem_chk and it did generate a log file , not knowing how long it would run I aborted after a few minute
it below here
 
Code: [Select]
[Pass] strlen("abc") => 0x3
[Pass] strlen("qwertyuiop") => 0xa
[Pass] strlen("") => 0x0
[Pass] strcpy(msg, "hi there") => 0x17b068
[Pass] msg => 'hi there'
[Pass] snprintf(a, sizeof(a), "foo") => 0x3
[Pass] snprintf(b, sizeof(b), "foo") => 0x3
[Pass] strcmp(a, b) => 0x0
[Pass] snprintf(a, sizeof(a), "bar") => 0x3
[Pass] snprintf(b, sizeof(b), "baz") => 0x3
[Pass] strcmp(a, b) => 0xfffffff8
[Pass] snprintf(a, sizeof(a), "Display") => 0x7
[Pass] snprintf(b, sizeof(b), "Defishing") => 0x9
[Pass] strcmp(a, b) => 0x4
[Pass] snprintf(buf, 3, "%d", 1234) => 0x2
[Pass] buf => '12'
[Pass] memcpy(foo, bar, 6) => 0x17b000
[Pass] foo => 'asdfghuiop'
[Pass] memset(bar, '*', 5) => 0x17afe0
[Pass] bar => '*****hjkl;'
       bzero32(bar + 5, 5)
[Pass] bar => '****'
       t0 = *(uint32_t*)0xC0242014 => 0xccf82
       msleep(250)
       t1 = *(uint32_t*)0xC0242014 => 0x61e8c
[FAIL] ABS(mod(t1-t0, 1048576)/1000 - 250) => 0x168
       LoadCalendarFromRTC( &now )
       s0 = now.tm_sec => 0x5
       Date/time: 2017/04/23 00:05:0ü&‚¸ ]&,½ l‚CK<&²Ã
–(¹`UrV‹x&’°ËX'¢½”k":‹%R­JD&ÿ½´j"‘K'¿Ëu%}©<L¢ ¨'²Ã w$Hªô^ÂÓ ì'‚½Ìv(mÇh‡BþŒP)¸J7%E¸ u’‹ÌØ(ÇKv& ½€wbÒ ¤&ÂØÒÓ2yßBÍс3ó8       m0 = MALLOC_FREE_MEMORY => 0x203c8
[Pass] p = _malloc(50*1024) => 0x104930
[Pass] CACHEABLE(p) => 0x104930
       m1 = MALLOC_FREE_MEMORY => 0x13bd8
       _free(p)
       m2 = MALLOC_FREE_MEMORY => 0x203d8
[Pass] ABS((m0-m1) - 50*1024) => 0x10
[Pass] ABS(m0-m2) => 0x10
       m0 = GetFreeMemForAllocateMemory() => 0x3e55e8
[Pass] p = _AllocateMemory(256*1024) => 0x972f94
[Pass] CACHEABLE(p) => 0x972f94
       m1 = GetFreeMemForAllocateMemory() => 0x3a5574
       -_FreeMemory(p)
       m2 = GetFreeMemForAllocateMemory() => 0x3e5580
[Pass] ABS((m0-m1) - 256*1024) => 0x74
[Pass] ABS(m0-m2) => 0x68
       m01 = MALLOC_FREE_MEMORY => 0x203d8
       m02 = GetFreeMemForAllocateMemory() => 0x3e5520
[Pass] p = _alloc_dma_memory(256*1024) => 0x409d0270
[Pass] UNCACHEABLE(p) => 0x409d0270
[Pass] CACHEABLE(p) => 0x9d0270
[Pass] UNCACHEABLE(CACHEABLE(p)) => 0x409d0270
       _free_dma_memory(p)
[Pass] p = (void*)_shoot_malloc(24*1024*1024) => 0x4e5d00e8
[Pass] UNCACHEABLE(p) => 0x4e5d00e8
       _shoot_free(p)
       m11 = MALLOC_FREE_MEMORY => 0x203d8
       m12 = GetFreeMemForAllocateMemory() => 0x3e6634
[Pass] ABS(m01-m11) => 0x0
[FAIL] ABS(m02-m12) => 0x1114
[FAIL] p = (void*)_shoot_malloc(24*1024*1024) => 0x0
[FAIL] UNCACHEABLE(p) => 0x40000000
[FAIL] ABS(m02-m12) => 0x101fc
4ÐD-òûÏÅ+fðع¤ŽØ*"åN¡)ׄŽâÍ@(ÒË z'vÂ<gÎ ¼%¢·KT$4­üE"¸Jô#B§Ê;#J <1"߉˜"Rœ‰'"©—by‰ ¡oÇÂo(À¡ýF!pÀp ¿ÇpÇÁpÃÑDApÅoLÄ¡ ÇáoGÃp<ÅaGLq‡Ä pÊ ‡`ÁqGÇq¬Ë‡ QqÌp¤Ì¡ÇÀqÏq¸Ì±GŒQq‡Ì q´ÉAÇhqpÇÈq”ɱX1qGÆ qdɇ|ÁpÉpÀÈ¡G˜1p‡ÊqÈËA‡¼QqÏqρGqrÇÐ)r@Õ¡(‡xtÛ9rœÛñ[Pass] suite = shoot_malloc_suite_contig(0) => 0x86b91c
[Pass] suite->signature => 'MemSuite'
[Pass] suite->num_chunks => 0x1
[Pass] suite->size => 0x1cf8000
[Pass] chunk = GetFirstChunkFromSuite(suite) => 0x86b940
[Pass] chunk->signature => 'MemChunk'
[Pass] chunk->size => 0x1cf8000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x42004070
[Pass] UNCACHEABLE(p) => 0x42004070
       shoot_free_suite(suite); suite = 0; chunk = 0;
[Pass] suite = shoot_malloc_suite(64*1024*1024) => 0x9b8758
[Pass] suite->signature => 'MemSuite'
[Pass] suite->num_chunks => 0x3
[Pass] suite->size => 0x4000000
[Pass] chunk = GetFirstChunkFromSuite(suite) => 0x9b877c
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x1cf8000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x42004070
[Pass] UNCACHEABLE(p) => 0x42004070
       chunk = GetNextMemoryChunk(suite, chunk) => 0x9b87f4
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x3724000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x565d00e4
[Pass] UNCACHEABLE(p) => 0x565d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0x9b886c
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x4000000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x525d00e4
[Pass] UNCACHEABLE(p) => 0x525d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0x0
[Pass] total => 0x4000000
       shoot_free_suite(suite); suite = 0; chunk = 0;
[FAIL] suite = shoot_malloc_suite(0) => 0x0
[FAIL] suite->signature => '‡'
[FAIL] suite->num_chunks => 0xe59ff014
t*†H½.½°ü"¶ˆ+²…H­-†ŒXþÒÍH0,BˆÞ-ֈظ’– -ŽHú,ǃ¨•ÒX˜,"‰ˆÛ1÷‘hüÊH˜/ãŽH +´ŒÔ$ÉÄ/‹Èâ/ÓLgŠ”A¤Á bB#¾¼É”VL€@³¦In.ˆÔu’ÔØ#"z‡-"œx wÀ quÇöHu¼’[GœuÇúEu˜ùEpAtGúTtöQ=ÇÑsí:v¨þLÇä!u‡þ Zu<õ1IÇ8¡tÇî6sÌîaF‡Ôs‡ì4s íá2‡ ±rGè;t ðAFÇh¢t‡  UuûA[X "w$#¤}ìVú‡L'ƒH™)@‡àÌ‚Hð-‚‡H¡(8‚X`
HÔ&²ƒÈ,­‘¸
$6£Ê¬@¿µF$„‹àE¶Ê9<Ò ¬QC}É`3”/ðސýRáH,0C‰ /‹„ê±HÈ-"ŠÖ.¼Œæ2͐0òŒï1è’#ÃÉ|1ˆî,•‰³k€)r‚ÇW!³uDöÑF‡1sÇç@s0äAE‡$ Rv&¹…üÆB–ˆ -âˆHÑ.ÈŒðòÊÈì/cˆ
0ûŽ ÃíȨ/R‹ÈÓ.¸‹¸ãÒ刔0ãI$9œ«˜+ô „F¤»KfD³·T44{Ë@D”º pG÷ÀTsì °D3´
Þ6t˜,úBÈHˆ("GC Žuðïá6Œs‡ïRvÜâžG#R{Ç*"¯y BfÇÈÑu‡õSu
ò‡Ü#r€Hƒ.£˜có’Ê\?”ºÌ€MžÓx %YÍ|OäÎ̯DQ¶ÅCpÊ63š ,$Š ›wˆ(‚†‡)ŒìÏ@I\2ƒ¡Š˜AÜ¿‡”㠐PÕÜNfY{ìˆìÕCÏÔ`¦ö4c×ûœXö8Ég¦¢i–é‡ÖF hFЋgI éb†ûfæÿyg'Ù‚Ö=ñgŠg'AtV#ÐagcfþxoF+ÐqgÆqfþ$CæÐˆcÆõ
^0ë˜Ë5¾ÎX\ÅìNÈ\·ì€Íµ¸h\•êµZ…å(•>ÎÌWâNrVßl`µë¼UÅávUÜÌ@žlSÕÙÍ*S•Ùu…
°QµÖÍQlÕP …;ͰNôÎLÊJËÇx‘4B ,F´¾KBB±·´/K\>Ó°Ëò@E¸|ŠôÈÍa— “†‹TR1wÉný‰cÔ‰ƒä……7L >Ã©Š¥;¥­8ÚÓ ?”² ?9³°õC! `>±ÊÚ=ê¬4Å“»Š <3«Š¼;¯«|¶¤Êt;cªŠ¹;¬«È¸3¤Š€;“§J¶: ªô³cªÊ$;é
¨9¦ä—ãrÊÐ8“¥Êˆ8h¥Ìa
 :Ó¥
¡9s¦<›ƒhJD9S¥Ê‡8W¥ “3e
9ó¥
™9l§À ³zJ:㨊¬9—¨ø£C›ŠL:3¨Š¤:†¨€ “Šü9“§Šš9h§l“CgJ 9¦
‹8h¥èŠã]
 8¦
8p¥À‰cŠÌ8c¦Š’8f¦¼‰SLЏ7s£Š}60 ¬bcŠT5£‰N6ô£,гœ
´<ä²
DÕÇôŠÔûŒ°L4×ÍãN”Ù$Ýd¤ÌM”×MÝMƒ×Tٌ̈́M”ØMÐLØÌÓ¤˜ÍÄMtØ
ØM˜×Ñ”c¼L”ÙÍÓLy×tÏ´ƒÈLtÖMÇLƒÖlÁTnÍxKՍ¿KEÔä¯4/ͬJdԍ§J+Ïp´üŒxHÄÍ ~HàÌ}ÄÃŒ4G„ËÌnF±Êäe”Ì(FÆLTC6ÁÐ'4ó‹°Aƒ½‹ý>ž¶¸×#@ `<3°J¤9½¨sãVŠ(6s¢IJ3ô›`£’ H0•Ií.>Í"ßHˆ+R‹ª)§ˆÀ‹‚l (…È{&E‚üf’pGÀpÔÀ¡û† !p‡Á
o¾¡
±pGÁpì¾AG$1pÇÄpøÄ¡Ç$1qǽpÁÑG$qqÃr8Á‡8±rÃ(rd‘%h1sÃ/r,ÅÑ)‡PÑq‡Ã'rhÁÑ)‡8‘rGÄ-r4Ʊ*‡ParÇÄ&rLÃ%‡`¡qÂq@Á‡$!r‡À#rHÂ1‡tsÅ*rDÅÁ;ÇxasGÉBsTÃ!<„QtGÌRuÇñe‡œáuÇÌav¸È1\G|ñuÈUu È1EÇœtÅ/rhÇÑ(G°‘tËVyÓa¹ÇÌá~Hå6‰¼öÁÐHÄrI X™äÒËÉ !¢œ !ᝀÌIŒ!B‰!âœ|"·‰!ÒšI m“Œ
úHL RH áp rú!’” $#½¡lEB…Š8%R®Ka& ±pb20‹Ô&² l&B³¸nâC‹'bµ u'Iµq"j 0'²¶ z'o¸|zbË”'·K~'²·Ü~ò›‹ü'2¹ (¼»ˆ¸Kh(òº‹†(Á»D…R¼‹€(R½‹Š(î¾ÀŠ’æ‹,)r¿L‘(Áä–â h)RÁÌ–))ÂX“’3ŒŒ)RÄL)\Äl ’XŒä)bÅÌŸ*iÇ £’‚ ˜*rÈŒ§*œÉD¨R£Ìh*‚É̪*ÀÌ„®bÃÌä*rÌ ³+ÛÍ ³‚èL8+bÍ̵+éвRç̼+²ÎŒÂ+ÿÑ„·R0,ÂÑͼ+3ÔôÄ\
D,BÕMÇ,\ÖtËUÍ-R×ÍË,’ØìÏR¥M(-òٍÐ-¼Ü4ÙÒ×
”-òۍÓ-àݔׂëÍø-Þ
Ü-üàüáBøÍ(.’àÎç.'Ý ÌM¨+2Ð̤(¼<jRúJˆ$ò¢I* `Š„ô±Jˆ€ñ‚Èé:ŠìÿPIX!B¡J*"ž®ø7rVK0%ÒÅ       SetGUIRequestMode(0); msleep(1000);
[FAIL] CURRENT_DIALOG_MAYBE => 0x26
[FAIL] display_idle() => 0x0
There some memory address stuff but not LV , so I should re-run the stub test and see  if there any LV stuff ?
is there any info I can read on how to use the mem_spy for the raw lv buffer or is this a red herring
thanks in advance.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 24, 2017, 05:15:56 AM
Ok run the full stubs test with "new-lv-buffer-detections" branch results below
Code: [Select]
[Pass] strlen("abc") => 0x3
[Pass] strlen("qwertyuiop") => 0xa
[Pass] strlen("") => 0x0
[Pass] strcpy(msg, "hi there") => 0x177048
[Pass] msg => 'hi there'
[Pass] snprintf(a, sizeof(a), "foo") => 0x3
[Pass] snprintf(b, sizeof(b), "foo") => 0x3
[Pass] strcmp(a, b) => 0x0
[Pass] snprintf(a, sizeof(a), "bar") => 0x3
[Pass] snprintf(b, sizeof(b), "baz") => 0x3
[Pass] strcmp(a, b) => 0xfffffff8
[Pass] snprintf(a, sizeof(a), "Display") => 0x7
[Pass] snprintf(b, sizeof(b), "Defishing") => 0x9
[Pass] strcmp(a, b) => 0x4
[Pass] snprintf(buf, 3, "%d", 1234) => 0x2
[Pass] buf => '12'
[Pass] memcpy(foo, bar, 6) => 0x176fe0
[Pass] foo => 'asdfghuiop'
[Pass] memset(bar, '*', 5) => 0x176fc0
[Pass] bar => '*****hjkl;'
       bzero32(bar + 5, 5)
[Pass] bar => '****'
       t0 = *(uint32_t*)0xC0242014 => 0x63161
       msleep(250)
       t1 = *(uint32_t*)0xC0242014 => 0xa1441
[Pass] ABS(mod(t1-t0, 1048576)/1000 - 250) => 0x4
       LoadCalendarFromRTC( &now )
       s0 = now.tm_sec => 0x15
       Date/time: 2017/04/23 18:34:21
       msleep(1500)
       LoadCalendarFromRTC( &now )
       s1 = now.tm_sec => 0x17
[Pass] mod(s1-s0, 60) => 0x2
[Pass] mod(s1-s0, 60) => 0x2
       m0 = MALLOC_FREE_MEMORY => 0x203d8
[Pass] p = _malloc(50*1024) => 0x104930
[Pass] CACHEABLE(p) => 0x104930
       m1 = MALLOC_FREE_MEMORY => 0x13bd8
       _free(p)
       m2 = MALLOC_FREE_MEMORY => 0x203d8
[Pass] ABS((m0-m1) - 50*1024) => 0x0
[Pass] ABS(m0-m2) => 0x0
       m0 = GetFreeMemForAllocateMemory() => 0x3eb7fc
[Pass] p = _AllocateMemory(256*1024) => 0x9670fc
[Pass] CACHEABLE(p) => 0x9670fc
       m1 = GetFreeMemForAllocateMemory() => 0x3abaa0
       -_FreeMemory(p)
       m2 = GetFreeMemForAllocateMemory() => 0x3ebaac
[Pass] ABS((m0-m1) - 256*1024) => 0x2a4
[Pass] ABS(m0-m2) => 0x2b0
       m01 = MALLOC_FREE_MEMORY => 0x203d8
       m02 = GetFreeMemForAllocateMemory() => 0x3eba4c
[Pass] p = _alloc_dma_memory(256*1024) => 0x4096713c
[Pass] UNCACHEABLE(p) => 0x4096713c
[Pass] CACHEABLE(p) => 0x96713c
[Pass] UNCACHEABLE(CACHEABLE(p)) => 0x4096713c
       _free_dma_memory(p)
[Pass] p = (void*)_shoot_malloc(24*1024*1024) => 0x42004074
[Pass] UNCACHEABLE(p) => 0x42004074
       _shoot_free(p)
       m11 = MALLOC_FREE_MEMORY => 0x203d8
       m12 = GetFreeMemForAllocateMemory() => 0x3ebaac
[Pass] ABS(m01-m11) => 0x0
[Pass] ABS(m02-m12) => 0x60
[Pass] suite = shoot_malloc_suite_contig(24*1024*1024) => 0xa6e5e4
[Pass] suite->signature => 'MemSuite'
[Pass] suite->num_chunks => 0x1
[Pass] suite->size => 0x1800000
[Pass] chunk = GetFirstChunkFromSuite(suite) => 0xa6e608
[Pass] chunk->signature => 'MemChunk'
[Pass] chunk->size => 0x1800000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x42004070
[Pass] UNCACHEABLE(p) => 0x42004070
       shoot_free_suite(suite); suite = 0; chunk = 0;
[Pass] suite = shoot_malloc_suite_contig(0) => 0xa7a038
[Pass] suite->signature => 'MemSuite'
[Pass] suite->num_chunks => 0x1
[Pass] suite->size => 0x1cf8000
[Pass] chunk = GetFirstChunkFromSuite(suite) => 0xa7a05c
[Pass] chunk->signature => 'MemChunk'
[Pass] chunk->size => 0x1cf8000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x42004070
[Pass] UNCACHEABLE(p) => 0x42004070
       shoot_free_suite(suite); suite = 0; chunk = 0;
[Pass] suite = shoot_malloc_suite(64*1024*1024) => 0xa7a69c
[Pass] suite->signature => 'MemSuite'
[Pass] suite->num_chunks => 0x3
[Pass] suite->size => 0x4000000
[Pass] chunk = GetFirstChunkFromSuite(suite) => 0xa7a6c0
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x1cf8000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x42004070
[Pass] UNCACHEABLE(p) => 0x42004070
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa7a738
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x3724000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x5a5d00e4
[Pass] UNCACHEABLE(p) => 0x5a5d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa7a7b0
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x4000000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x565d00e4
[Pass] UNCACHEABLE(p) => 0x565d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0x0
[Pass] total => 0x4000000
       shoot_free_suite(suite); suite = 0; chunk = 0;
[Pass] suite = shoot_malloc_suite(0) => 0xa89580
[Pass] suite->signature => 'MemSuite'
[Pass] suite->num_chunks => 0x7
[Pass] suite->size => 0xb700000
[Pass] chunk = GetFirstChunkFromSuite(suite) => 0xa895a4
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x1cf8000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x42004070
[Pass] UNCACHEABLE(p) => 0x42004070
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa8961c
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x3724000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x5a5d00e4
[Pass] UNCACHEABLE(p) => 0x5a5d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa89694
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x5150000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x565d00e4
[Pass] UNCACHEABLE(p) => 0x565d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa8970c
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x6b7c000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x525d00e4
[Pass] UNCACHEABLE(p) => 0x525d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa89784
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x85a8000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x4e5d00e4
[Pass] UNCACHEABLE(p) => 0x4e5d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa897fc
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0x9fd4000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x4a5d00e4
[Pass] UNCACHEABLE(p) => 0x4a5d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0xa89874
[Pass] chunk->signature => 'MemChunk'
[Pass] total += chunk->size => 0xb700000
[Pass] p = GetMemoryAddressOfMemoryChunk(chunk) => 0x465d00e4
[Pass] UNCACHEABLE(p) => 0x465d00e4
       chunk = GetNextMemoryChunk(suite, chunk) => 0x0
[Pass] total => 0xb700000
       shoot_free_suite(suite); suite = 0; chunk = 0;
       EngDrvOut(LCD_Palette[0], 0x1234)
[Pass] shamem_read(LCD_Palette[0]) => 0x1234
       call("TurnOnDisplay")
[Pass] DISPLAY_IS_ON => 0x1
       call("TurnOffDisplay")
[Pass] DISPLAY_IS_ON => 0x0
       call("TurnOnDisplay")
[Pass] DISPLAY_IS_ON => 0x1
       SetGUIRequestMode(1); msleep(1000);
[Pass] CURRENT_DIALOG_MAYBE => 0x1
       SetGUIRequestMode(2); msleep(1000);
[Pass] CURRENT_DIALOG_MAYBE => 0x2
       SetGUIRequestMode(0); msleep(1000);
[Pass] CURRENT_DIALOG_MAYBE => 0x0
[Pass] display_idle() => 0x1
       GUI_Control(BGMT_PLAY, 0, 0, 0); msleep(500);
[Pass] PLAY_MODE => 0x1
[Pass] MENU_MODE => 0x0
       GUI_Control(BGMT_MENU, 0, 0, 0); msleep(500);
[Pass] MENU_MODE => 0x1
[Pass] PLAY_MODE => 0x0
[Pass] MEM(dialog->type) => 0x414944
       GUI_Control(BGMT_MENU, 0, 0, 0); msleep(500);
[Pass] MENU_MODE => 0x0
[Pass] PLAY_MODE => 0x0
       task_create("test", 0x1c, 0x1000, test_task, 0) => 0x818400b4
[Pass] test_task_created => 0x1
[Pass] get_task_name_from_id(get_current_task()) => 'run_test'
[Pass] mq = mq ? mq : (void*)msg_queue_create("test", 5) => 0x81940086
[Pass] msg_queue_post(mq, 0x1234567) => 0x0
[Pass] msg_queue_receive(mq, (struct event **) &m, 500) => 0x0
[Pass] m => 0x1234567
[Pass] msg_queue_receive(mq, (struct event **) &m, 500) => 0x9
[Pass] sem = sem ? sem : create_named_semaphore("test", 1) => 0x81e6015a
[Pass] take_semaphore(sem, 500) => 0x0
[Pass] take_semaphore(sem, 500) => 0x9
[Pass] give_semaphore(sem) => 0x0
[Pass] take_semaphore(sem, 500) => 0x0
[Pass] give_semaphore(sem) => 0x0
[Pass] rlock = rlock ? rlock : CreateRecursiveLock(0) => 0x82360098
[Pass] AcquireRecursiveLock(rlock, 500) => 0x0
[Pass] AcquireRecursiveLock(rlock, 500) => 0x0
[Pass] ReleaseRecursiveLock(rlock) => 0x0
[Pass] ReleaseRecursiveLock(rlock) => 0x0
[Pass] ReleaseRecursiveLock(rlock) => 0xf
[Pass] f = FIO_CreateFileEx("test.dat") => 0x4
[Pass] FIO_WriteFile(f, (void*)ROMBASEADDR, 0x10000) => 0x10000
[Pass] FIO_WriteFile(f, (void*)ROMBASEADDR, 0x10000) => 0x10000
       FIO_CloseFile(f)
[Pass] FIO_GetFileSize("test.dat", &size) => 0x0
[Pass] size => 0x20000
[Pass] p = alloc_dma_memory(0x20000) => 0x409648dc
[Pass] f = FIO_Open("test.dat", O_RDONLY | O_SYNC) => 0x4
[Pass] FIO_ReadFile(f, p, 0x20000) => 0x20000
       FIO_CloseFile(f)
       free_dma_memory(p)
[Pass] count => 0x3a98
[Pass] FIO_RemoveFile("test.dat") => 0x0
       SW1(1,100)
[Pass] HALFSHUTTER_PRESSED => 0x1
       SW1(0,100)
[Pass] HALFSHUTTER_PRESSED => 0x0

lots of address but doesn't look like any LV buffer address etc. .... unless someone smarter can help .

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2935%2F33390807224_e20c28e94c.jpg&hash=890159cbceb1a74fcf7b7f981e6137cf) (https://flic.kr/p/SSCBEQ)
Screenshot EDMAC (https://flic.kr/p/SSCBEQ) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Screenshot of the edmac  and I notice that edmac#5 is now active (green) (3570x1267) that the buffer size I used with very limited success to get 3584x1068 in post #27
When I did update to this branch I notice it was on the older ml core even thou I cloned a fresh copy of the source.
on the newer ml source code that channel is not active (yellow) so I have to wonder why ? could this be a issue , have to investigate further
Code: [Select]
# Magic Lantern v2.3.NEXT.2017Apr23.5D2212 (09dcd0475cb3+ (new-lv-buffer-detection))
# Built on 2017-04-23 05:47:00 UTC by ml@ml-pc
# Configuration saved on 2017/04/23 18:44:11
beta.warn = 23

So I'm not sure it these stubs are still useful .

Next I complied the "allocate-raw-lv-buffer" and that one seem to use the newer ml core source .
 
Code: [Select]
# Magic Lantern Nightly.2017Apr24.5D2212 (5d168ad4e8c9+ (allocate-raw-lv-buffer))
# Built on 2017-04-24 01:16:11 UTC by ml@ml-pc
# Configuration saved on 2017/04/23 19:47:45
beta.warn = 23

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F3%2F2885%2F34232210575_4191fe7346.jpg&hash=ff42097e92632f2e9668cc453ba786b1) (https://flic.kr/p/U9Z2za)
Screenshot of Lv with MPU_Dump running (https://flic.kr/p/U9Z2za) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

So I found this very interesting as I pushed the 5x Zoom button I notice that the Live view resolution is not what I thought  & have read to be  ???
In 1:1 a surprising 2040x1268  and in 3x crop (5x Zoom) 2312x1128 , Really !
I was under the impression that full Liveview was only 1880x1250  and 3x crop was 2152x1078.
Wondering out loud , could be a result of the edmac channel #5 being active (maybe  larger buffer size)
More mysteries to solve  :D
I have a MPU-EEP , Ram & Rom dumps not sure how to use them yet , there may be nothing useful either .

On a lighter note I never notice the False Color Overlays , there where on by default in the  "Magic Lantern v2.3.NEXT" version
If it wasn't for that i would have never notice them  8)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F5%2F4156%2F33390807244_93b069408b.jpg&hash=21b9db778f72b1e2aaffe1d9ebba9b48) (https://flic.kr/p/SSCBFb)
False Color Overlays  (https://flic.kr/p/SSCBFb) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on April 24, 2017, 09:06:56 AM
@reddeercity I've been watch your God like devotion to further improving 5D2 Thanks a million. This looks promising so when can we mere mortals get to test it?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on April 24, 2017, 09:42:14 AM
[emoji2]
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Greg on April 24, 2017, 02:15:04 PM
In 1:1 a surprising 2040x1268  and in 3x crop (5x Zoom) 2312x1128 , Really !
I was under the impression that full Liveview was only 1880x1250  and 3x crop was 2152x1078.

This is probably a resolution with black borders.

raw.c
Code: [Select]
#ifdef CONFIG_5D2
skip_top        = zoom ?   52 : 18;
skip_left       = 160;
#endif
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: PressureFM on April 24, 2017, 06:04:24 PM
How can people with 5D2 help?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 25, 2017, 05:09:21 AM
@PressureFM  thanks , the more the merrier .
Currently the issue is the customs Raw LV buffer , I've been trying to spy the memory to find the LV raw buffer
and looking thought my 5d2 decompiled rom dump for addresses , a1ex gave me some clues at the links
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183374#msg183374
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183382#msg183382

I found the registers needed to modify frame size with adtg_gui but the problem is the LV buffer
I did have some success with my  RAW_LV_BUFFER_ALLOC_SIZE (2040*1267) as per post #27
The reason I used that was it was the largest buffer on the edmac & it didn't change when switch to Photo Liveveiw see post #49
Edmac#5 buffer size is what I used , now I think there's I little dilemma here , when I updated to "new-lv-buffer-detections" branch and complied
I ended up with the older core "Magic Lantern v2.3.NEXT" , reason I mention this edmac#5 is Green (active) where in the nightly builds compiles
on newer code edmac Channel #5 is Yellow , which I think is a error  or all this could mean nothing , just thought I would mention it .
So Yea LV raw buffer needs to be solved & I thing that will also help to get 10 & 12bit problem in 1:1 solved also

Link to my bitbucket downloads for those memspy builds I talked about .
magiclantern-Nightly.memspy.lv.raw.buffer.extended.resoultion.2017Apr24.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-Nightly.memspy.lv.raw.buffer.extended.resoultion.2017Apr24.5D2212.zip)
magiclantern-v2.3.NEXT.Memspy.2017Apr23.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-v2.3.NEXT.Memspy.2017Apr23.5D2212.zip)
You may just want to compile your own copy of adtg_gui from the ISO-research branch or you can download a copy from my bitbucket download.
So any suggestion are very welcome  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 25, 2017, 07:02:24 AM
Code: [Select]
ff036750 lv_rshd_raw
ff0365f4 lv_raw_dump
ff036704 lv_save_raw
ff036724 lv_continous_frame
ff036760 lv_rec
ff03676c lv_output_device
ff036604 lv_continous_frame_save
ff036784 lv_hd
some lv raw stuff from my decompiled rom dump , I think the ff036750 lv_rshd_raw is what I'm after
Code: [Select]
FF036520: e59f1224  ldr r1, [pc, #548] ; 0xff03674c: pointer to 0xff8347f8 ⬁
FF036524: e28f0f89  add r0, pc, #548 ; *'lv_rshd_raw'

dropbox link to FF036524.htm (https://www.dropbox.com/s/8xvupbnn6oa5vb7/0xFF036000%20___%200xFF037FFF_htm%23_FF036524.htm?dl=0) where I found the raw liveview from the decompiled rom
Title: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on April 25, 2017, 10:40:27 AM
This is all nothing short of spectacular and wonderful progress so far. Just confirmed w my co-worker to bring his 5D2 to work tmw for me to test out the memspy build.

Thanks @reddeercity just for being you, as always!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on April 26, 2017, 01:52:41 AM
@PressureFM  thanks , the more the merrier .
Currently the issue is the customs Raw LV buffer , I've been trying to spy the memory to find the LV raw buffer
and looking thought my 5d2 decompiled rom dump for addresses , a1ex gave me some clues at the links
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183374#msg183374
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183382#msg183382

I found the registers needed to modify frame size with adtg_gui but the problem is the LV buffer
I did have some success with my  RAW_LV_BUFFER_ALLOC_SIZE (2040*1267) as per post #27
The reason I used that was it was the largest buffer on the edmac & it didn't change when switch to Photo Liveveiw see post #49
Edmac#5 buffer size is what I used , now I think there's I little dilemma here , when I updated to "new-lv-buffer-detections" branch and complied
I ended up with the older core "Magic Lantern v2.3.NEXT" , reason I mention this edmac#5 is Green (active) where in the nightly builds compiles
on newer code edmac Channel #5 is Yellow , which I think is a error  or all this could mean nothing , just thought I would mention it .
So Yea LV raw buffer needs to be solved & I thing that will also help to get 10 & 12bit problem in 1:1 solved also

Link to my bitbucket downloads for those memspy builds I talked about .
magiclantern-Nightly.memspy.lv.raw.buffer.extended.resoultion.2017Apr24.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-Nightly.memspy.lv.raw.buffer.extended.resoultion.2017Apr24.5D2212.zip)
magiclantern-v2.3.NEXT.Memspy.2017Apr23.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-v2.3.NEXT.Memspy.2017Apr23.5D2212.zip)
You may just want to compile your own copy of adtg_gui from the ISO-research branch or you can download a copy from my bitbucket download.
So any suggestion are very welcome  :D

Forgive me for my ignorance but what do I do with the downloads to us this build?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 03, 2017, 08:41:09 AM
From the live view log , from DeafEyeJedi
Some very interesting information about liveveiw specially the "tomSetRawJpgMode (Type = 0x4)"
Thou I don't understand it all give more info to work from that I don't have before
thanks to a1ex  ;)

Code: [Select]
line 1760 --- 51BF1>      TOMgr:ffa21790:41:05:  tomSetRawJpgMode (Type = 0x4)
line 2163 --- 6A4E4>    Startup:ff8e5f88:98:03: GetImageTrimming X(2976)=0(2414, 2416), Y(1928)=0(1552, 1548)


line 2293 --- 72089> LiveViewMg:ff8e6108:98:02: Origin2 X=911, Y=518 W:1122 C:2976

line 2294 --- 721BA> LiveViewMg:ff8de8e0:9b:03: PROP_MOVIE_PARAM 0 1

line 2295 --- 722B0> LiveViewMg:ff8dd52c:9b:16: PROP_LIVE_VIEW_VIEWTYPE_SELECT 0->2

line 2296 --- 722F3> LiveViewMg:ff8dcdb0:9b:03: ViewType:2

line 4927 --- 38AAA>    Startup:ff867658:82:03: InitializeDisplayDeviceController (PUB)

start at line 4975
Code: [Select]
3BA12> GuiMainTas:ff892834:85:03: AllocateVramCBR pAddress=43f80000
3BA90> GuiMainTas:ff86503c:00:03: [BmpDDev] CreatePhysicalVram (PUB)
3C401> GuiMainTas:ff864db0:00:03: [BmpDDev] DisplayPhysicalScreen (PUB) 0x43F00008
3C49C> GuiMainTas:ff9a77d0:00:01: [CLKSAVER] ��ClockSave In��
3C4D9> GuiMainTas:ff864be8:82:02:  SelectParameterToBmp (PRI)
3C502> GuiMainTas:ff864a40:82:02:  SetParameterToBitmapDisplayDevice (PRI)
3C526> GuiMainTas:ff864d3c:82:02: EnableBitmapVBufferForPlayBackAndWaiting (PUB)

Code: [Select]
3D012> GuiMainTas:ff83f068:36:05:  RegisterAdapterStatusCallback
3DB69> GuiMainTas:ffa84814:04:02: Partial memcpy start (x,y,w,h)=( 0, 0, 960, 540 )
Code: [Select]
4039D> GuiMainTas:ffa84844:04:02: Partial memcpy end
403D0> GuiMainTas:ff9c17ac:83:03: GuiStartGraphics (PUB)
404CE> GuiMainTas:ff82ebd4:80:03: SRM_AllocateMemoryResourceForImgVram 131 3
40582>     RscMgr:ff8b6ea8:80:03: srmAllocateImgVram 131 3
405BD>     RscMgr:ff8b6ec8:80:03: ImgVram Before -1 -1 -1
405F8>     RscMgr:ff8b6fa4:80:03: ImgVram After 131 131 131
4067A> GuiMainTas:ffa41e6c:18:03: (PUB) StartImagePlayer(mode=0) 1203
406BE> GuiMainTas:ffa42000:18:01: mode:0 dec:0x0 rot:0x0 2nd:0x0
406E6> GuiMainTas:ffa4532c:18:03: (PUB) SetVisibleImageVramOffset 4064
40712> GuiMainTas:ffa42304:18:03: (PUB) SetCopyVramMode(mode=0) 1411
4073B> GuiMainTas:ff9c15b0:83:03: GuiGraphicsNormalMode (PUB)
Code: [Select]
40760> GuiMainTas:ffa4522c:18:03: (PUB) SetImageWorkMemory 1:0x40d00000, 2:0x41700000
40799> GuiMainTas:ffa45274:18:03: (PUB) SetImageWorkMaxPixel W:2200,H:1872,0x5e4340
407CD> GuiMainTas:ffa44db8:18:03: (PUB) SetEffectiveSizeOfJpeg(w:7488 h:4992) 2995
407F8> GuiMainTas:ffa44dd0:18:01: SetEffectiveSizeOfJpeg w:7488 h:4992
40824> GuiMainTas:ffa423f8:18:03: (PUB) SetYuvColorParameter 1447
4089A> GuiMainTas:ff9dc868:83:03: IDLEHandler INITIALIZE_CONTROLLER
408CD> GuiMainTas:ff9dd028:83:03: IDLEHandler GOT_TOP_OF_CONTROL
408F1> GuiMainTas:ff9c3ffc:83:03: GuiClearImage
40914> GuiMainTas:ffa420fc:18:03: (PUB) SetVramInformation(w:1920 h:1080)
4094C> GuiMainTas:ffa421cc:18:03: 1920x1080 BaseVram:1920, VramYuv:1
40977> GuiMainTas:ffa421cc:18:03: 1920x1080 BaseVram:1920, VramYuv:1
409A0> GuiMainTas:ffa42210:18:01: fLcd 0
Code: [Select]
409C8> GuiMainTas:ffa423b8:18:03: SetImageVramParameter x:0 y:0 w:1920 h:1080/* Aspect:0*/
409FE> GuiMainTas:ffa44c3c:18:03: (PUB) SyncroAllClearImagePlayWorkVramWithoutEngine 2953
40A27> GuiMainTas:ffa4161c:18:03: GetVramNumber 619
40A4B> GuiMainTas:ffa4161c:18:03: GetVramNumber 619
416A5> **INT-0Ah*:00095c38:00:00: >>> INT-Ah Timer ff81027c(0)
416E5> **INT-0Ah*:0000057c:00:00: <<< INT-Ah done
43DB3> **INT-0Ah*:00095c38:00:00: >>> INT-Ah Timer ff81027c(0)
43DDC> **INT-0Ah*:0000057c:00:00: <<< INT-Ah done
45389> GuiMainTas:ffa420fc:18:03: (PUB) SetVramInformation(w:720 h:480)
453B5> GuiMainTas:ffa421cc:18:03: 720x480 BaseVram:720, VramYuv:1
453DD> GuiMainTas:ffa421cc:18:03: 720x480 BaseVram:720, VramYuv:1
45406> GuiMainTas:ffa42210:18:01: fLcd 0
Code: [Select]
4542B> GuiMainTas:ffa423b8:18:03: SetImageVramParameter x:0 y:0 w:720 h:480/* Aspect:0*/
45459> GuiMainTas:ffa4161c:18:03: GetVramNumber 619
45483> GuiMainTas:ff863840:82:02: GetVramSize (PUB)
454BD> GuiMainTas:ff86413c:82:03: EnableImagePhysicalScreenParameter
4556B> GuiMainTas:ff9a77d0:00:01: [CLKSAVER] ��ClockSave In��
455AC> GuiMainTas:ff8638c4:82:01: ImgDDev SelectParameter DispType=0
455D1> GuiMainTas:ff863840:82:02: GetVramSize (PUB)
455FC> GuiMainTas:ff8640d4:82:02: EnableImageVBufferForPlayBackAndWait (PUB)
45688> GuiMainTas:ffa4161c:18:03: GetVramNumber 619
456BD> GuiMainTas:ffa4161c:18:03: GetVramNumber 619
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on May 14, 2017, 04:17:36 AM
Can somebody explain in laymans terms what needs to happen to get this to 550D.
And also what exactly is this. Is this for the different bit depths, or for the compressed raw, or for 1:1 pixels.
I'm never entirely sure.
In addition to that, is it compressed raw being referred to as "lossless".
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 15, 2017, 04:48:33 AM
Can somebody explain in laymans terms what needs to happen to get this to 550D.

The custom buffer is required, as ML redirects it only while recording.
In the crop_rec_4k branch I've used a SRM buffer (which can accommodate a full-res 14-bit picture).
This needs to be passed to SetEDmac as the first argument (instead of Canon's default buffer).
The buffer redirection without CONFIG_EDMAC_RAW_SLURP is fragile:
it relies on lucky timing. It's best refactored somehow, but cache patching is also ugly...
.... is it compressed raw being referred to as "lossless".
Yes
------------------------------------------------------------------------------------------------------------------------

I keep running in to walls .
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Forig15.deviantart.net%2F8f4f%2Ff%2F2010%2F294%2Fa%2Fa%2Fman_vs_wall___s_by_0vonix0-d318uuv.gif&hash=685b5e4916484266ca42c2473128519c)
What needed now is someone with advanced knowledge of digic4 Liveview buffers , that would be a main developer like a1ex
this is getting above my knowledge level right now.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on May 15, 2017, 07:13:53 AM
Ha ha. That's how I feel too. Been trying this on a couple of Digic V cameras but keep running into walls too.

Maybe put it aside and try something else for a while.


Sent from my iPad using Tapatalk
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Teamsleepkid on May 15, 2017, 10:30:59 AM
I want it so bad...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on May 15, 2017, 11:36:13 AM
For 550D, the interesting parts are:
- for video: the lossless compression
- for timelapse: the full-res LiveView

The first one was covered in detail in "the two in two out discussion", with partial success on 500D (only working outside LiveView, and output not decoded yet). Until the first issue (which wasn't present on 5D3) is solved, porting it probably doesn't make much sense, as the two models are very similar, and you'll likely run into the same problem. Newer models (60D and newer) are probably much easier to port, as their LiveView implementation is a lot closer to 5D3's.

The second one (full-res LiveView) requires adtg_gui (iso-research branch), but to explain in detail how it works, I need to try it myself on an old-gen camera, to have an idea of similarities and differences. The most promising attempts, at the time of writing, are on 700D (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183286#msg183286) and 5D2 (this thread).

However, increasing the resolution beyond what you get in regular LiveView requires allocating a new buffer. On recent cameras, it's easy (just enable some definitions and tell how much you need - see how it's done on 1100D and 5D3). On old generation cameras, it's hard and probably requires patching Canon code.

Other than that, it's just a matter of patching some registers (see the crop_rec source code for the various presets).

What needed now is someone with advanced knowledge of digic4 Liveview buffers

Unfortunately, my level of understanding is not there yet, but I've made some small progress emulating the LiveView in QEMU. On 500D, it appears to work for a few frames before locking up, so it's not yet published :D

However, both the 550D and 5D2 (and other models) now have the GUI emulation working (you can navigate Canon menu in the emulator), so the first step for LiveView is to get a set of MPU spells that covers... entering LiveView. This is easy - use the dm-spy-experiments branch (or the startup-log builds (https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/)), enter LiveView while the LED is blinking (that is, while debug messages are recorded) and then run the extract_init_spells.py (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/eos/mpu_spells/) script on the log file. This alone is enough for some LiveView bitmap overlays on 500D (Greg's screenshot (http://www.magiclantern.fm/forum/index.php?topic=2864.msg179867#msg179867) - didn't try on other models). Progressing from there is much harder. I can share the current patch if there is interest, but it's "too much of a hack" at this stage. I recommend starting with easier things instead.

Currently, adtg_gui and the QEMU LV trick are low-hanging fruits. Creating debugmsg.gdb files for camera models that don't have one, or adding more entries to an existing one, is also very easy (as they mostly contain stubs (http://www.magiclantern.fm/forum/index.php?topic=12177.0)).

Porting the FRSP emulation to QEMU ranges from extremely easy to hard, depending on model. For some models, you only need to supply a reference full-res DNG and to add the camera model to the test suite - in other words, confirming it's working. Others may require some minor fiddling, others may require a deeper level of understanding. To get started, look up FRSP or FA_CaptureTestImage on the QEMU test suite (https://builds.magiclantern.fm/jenkins/job/QEMU-tests/) (logs, screenshots). At the time of writing, this is working on 60D, 1200D, and also 5D3 1.1.3 (where my test image has a hot pixel in the OB area, which is the reason for not passing the test). Models with GUI emulation are probably easier to adapt (just a guess, as I didn't try).

Playing with custom logging (http://www.magiclantern.fm/forum/index.php?topic=2388.msg113154#msg113154) on the dm-spy branch is also easy once you get some courage, and can be done on both a real camera or QEMU. Understanding the logs is harder. LiveView is *very* complex, so it makes sense understanding smaller bits first (such as logging task names as they are started, or following a button event).

Debugging the lossless compression routine in QEMU should also be accessible. You'll need to run both Canon's original routine and your modified one with "-d io" and compare the JPCORE and related I/O activity - see Greg's examples (http://www.magiclantern.fm/forum/index.php?topic=18443.msg184283#msg184283).

Emulating a CR2 picture taking (until getting a valid CR2 on the SD/CF card image) should be also doable, and a whole lot easier than emulating LiveView. Maybe I should try that first.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on May 15, 2017, 01:00:59 PM
use the dm-spy-experiments branch (or the startup-log builds (https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/)), enter LiveView while the LED is blinking (that is, while debug messages are recorded) and then run the extract_init_spells.py (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/eos/mpu_spells/) script on the log file.
Does this still need to be done for 5D2?
just in case: https://www.dropbox.com/s/84hknim2jrhd5ww/copypasta.txt?dl=0 (https://www.dropbox.com/s/84hknim2jrhd5ww/copypasta.txt?dl=0)
The script needed some changes tho, this didn't work:
Code: [Select]
cam_dir = [d for d in os.listdir(ml_dir)
             if d.split(".")[0] == camera_model
             and os.path.isfile(os.path.join(ml_dir, d, "gui.h"))
          ][0]
just manually set it to "5D2.212", also default ML directory was wrong in my case.
^^ Maybe that helps anyone else doing it.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on May 15, 2017, 01:19:55 PM
Right now, none of the spells from the repository include LiveView activity (it was easier to understand how it works without it). I've only tried the LiveView trick on 500D (which I find easier to understand). So, if you want to play around, you can give it a try; however, once you get past the initial GUI (or maybe earlier), you *will* "run into walls". I'm currently in the middle of one such wall :P

An initial goal would be to get both the menu navigation and the LiveView trick in the same set of spells - that didn't work in my experiment, but it's probably easy to fix. Either perform both menu navigation, play mode switch and LiveView in the same logging session to re-create the spell set from scratch, or figure out which bits are specific to LiveView and add them to current spell set.

BTW, the script is meant to be run from the QEMU installation directory (outside the ML tree). That's probably the reason you've got the path error.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 16, 2017, 07:35:50 AM
Thanks a1ex , I was hoping for something like this  :D gives me more meat to chew on.
This alone is enough for some LiveView bitmap overlays on 500D (Greg's screenshot (http://www.magiclantern.fm/forum/index.php?topic=2864.msg179867#msg179867) - didn't try on other models).
Progressing from there is much harder. I can share the current patch if there is interest
I would be interested in at least looking at the code .
Great there's still a change this can be ported  8)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on June 12, 2017, 02:17:53 PM
Any progress on this? Seems to have stopped still for a month.
Has the way to crop record been discovered yet?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on June 19, 2017, 06:56:49 AM
Any progress on this? Seems to have stopped still for a month.
No.
Reason , there is no interest in any Digic IV (4) development The forum is being driven by Digic V (5) and up cameras , in particular 5D3.
If you think I'm kidding just look at all the posts over the last mouth . In fact even 10 or 12bit development is dead for Digic IV cam's
other then the experimental builds I have on my bitbucket download (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/)  , no one else really care and that a trouble shame ! :(
As I don't see anyone else participating here , I can no longer justify developing this thread .

Has the way to crop record been discovered yet?
It's very close (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183098#msg183098) , the only issue is having a custom image buffer with the right address .
I already know which registers that need to modified but with out the correct buffer/memory address it stop there .

Even thou I would really like to see this for the 5d2 , I want this for all digic 4 cams ( Wishful thinking)
In the near future if I see activity here then I my continue further , until such time I'll just make improvement's for myself .
Sorry
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on June 19, 2017, 10:42:53 AM
I'm very keen for it for 550D. Hopefully to have something that can record 1080p 10bit.
Yeah the discussions on the forums do seem to be around 5d3, but I still think there's interest for Digic IV, they're just not very talkative.
And I think it's also partly due to it working on Digic 5, so people are using it. Where as it doesnt work on 4.
I can imagine Digic 4s becoming popular in discussion if there were things like 2K/4K 10bit and all that hoohar.

Let me know if I can do anything.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Ottoga on June 19, 2017, 02:06:48 PM
+1 here with a 7D. Can't code but can test.

I'm sure that there are plenty of lurkers waiting for something exciting to happen that they can assist with.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: loknar on June 19, 2017, 08:54:30 PM
@reddeercity
It's not like we wouldn't like new builds, but most of us are not programmers, so we do some test, and that's it. I for one would love 10-bit RAW working on 550D, but I have same problems as others, all frames are corrupted 1/3 of screen is shifted and frozen. Since this has been reported, i'm afraid that's all I can do.

(I was also hoping in mv1080 on EOS M, but all my questions in appropriate threads have been ignored, so i kinda gave up)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on June 20, 2017, 01:22:56 AM
(I was also hoping in mv1080 on EOS M, but all my questions in appropriate threads have been ignored, so i kinda gave up)

What do you mean? EOS M can do mv1080 if you turn on H.264 proxy. It is rather buggy and experimental but hey.

http://www.magiclantern.fm/forum/index.php?topic=16608.msg179969#msg179969
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Tullen on June 20, 2017, 04:25:40 PM
I really want to get this going, especially for the 50D. As soon as I have fished my thesis I hope to be able to help. Even would like to start a Magic Lantern club in Stockholm. That would be in September though. Please dont give up. I am sure that more people would like to help. I am willing to put in time for this for sure.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: loknar on June 20, 2017, 08:59:34 PM
What do you mean? EOS M can do mv1080 if you turn on H.264 proxy. It is rather buggy and experimental but hey.

http://www.magiclantern.fm/forum/index.php?topic=16608.msg179969#msg179969

I mean I wasn't able to replicate it, I wasn't able to get same nightly and with my version of ML although i've been able to capture whole frames, all were in pink and after correction of black and white levels it has been extremely noisy to the point of unusability. Same processing of test mlv of Koks the cat lead to excellent results, just my camera didn't deliver same input. When I asked about what build i should use, it stayed unanswered  ( http://www.magiclantern.fm/forum/index.php?topic=9741.msg183234#msg183234 ) (not complaining, just stating). I've got in bookmarks your bitbucket page, but you wrote that your version was really experimental, and also i'd have to update firmware to 2.03, and i thought since 2.02 is more mainstream, that development would go faster there.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on June 26, 2017, 06:08:01 AM
Ok --So I decided to forge ahead after urging from some forum members and help from a1ex.
So to get up to speed read the posts below to those members who are following the development , should help explain where I am in all this.
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186209#msg186209
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186213#msg186213
Basically I trying to get back to the 3584x1068 raw buffer I had working here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183098#msg183098)
Stay tuned  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on June 26, 2017, 02:12:00 PM
Good to hear! I think many people are ready to step up if you need a hand.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: nikki on June 26, 2017, 09:11:55 PM
let us know I have my 5d2 ready and charged
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Tullen on June 27, 2017, 01:08:43 PM
Ok --So I decided to forge ahead after urging from some forum members and help from a1ex.
So to get up to speed read the posts below to those members who are following the development , should help explain where I am in all this.
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186209#msg186209
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186213#msg186213
Basically I trying to get back to the 3584x1068 raw buffer I had working here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183098#msg183098)
Stay tuned  :)

Great to hear. As I said, I will be helping as much as I can from September onward.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: scotophorus on June 28, 2017, 12:17:07 AM
Ok --So I decided to forge ahead after urging from some forum members and help from a1ex.
So to get up to speed read the posts below to those members who are following the development , should help explain where I am in all this.
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186209#msg186209
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186213#msg186213
Basically I trying to get back to the 3584x1068 raw buffer I had working here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183098#msg183098)
Stay tuned  :)

This is great news. Big thanks to you, Alex, and all the other geniuses for pushing forward the development. I'm ready to test and help in anything you need.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on June 28, 2017, 06:41:46 AM
Great to hear. As I said, I will be helping as much as I can from September onward.
Sounds good , In the mean time I would suggest familiarize yourself with this thread , 10-12bit thread and of course the  iso-research (https://bitbucket.org/hudson/magic-lantern/src/bd8e89d9dccf2d1541bb449efd06bb450cdce153/?at=iso-research) source code. (That's where I started from)
You will need a developing environment -- I use a VM (VirtualBox) on Window7 Pro with a preconfigured magic lantern clone of Ubuntu (http://www.magiclantern.fm/forum/index.php?topic=18259.msg174934#msg174934) ready to compile ML . Download links for clone of Ubuntu  here (http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989)
You can run the VM under Mac or Windows just get the correct version for your OS , if you feel adventurous you can setup your own environment @dfort has make some tutorial here (http://www.magiclantern.fm/forum/index.php?topic=16012.msg155422#msg155422) and here (http://www.magiclantern.fm/forum/index.php?topic=15894.msg154435#msg154435)
You need to be very comfortable with compiling , merging , update source with "HG" etc. ..... I work with a VM because it's faster & easier to get up and running in less then 30 mins .

 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: whitelight on July 10, 2017, 12:53:18 AM
Hello, I just want to thank all the developers for the work done, I wish this will one day be ported to 7D, old camera but still doing an amazing job when used with ML.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: flofifull on July 10, 2017, 11:53:08 AM
Hello, I just want to thank all the developers for the work done, I wish this will one day be ported to 7D, old camera but still doing an amazing job when used with ML.

+1
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on July 17, 2017, 04:12:43 AM
5d2 users --could I please get a screenshot of the edmac in video mode like in this post here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183621#msg183621) I what to compare to my 5d2 , in fact it wouldn't hurt to see other digic iv cams also for reference , Thanks  :) 

Edit:
You can use the latest nightly , but for the 5d2 users could you do 2 screenshot , 1 with the latest nightly and the other with the adtg_gui build from my bitbucket download  here (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-adtg_gui.2017Apr04.5D2212.zip)  please
screen shoot feature in the debug menu , you have 10's to get to the right window.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Tullen on July 17, 2017, 01:31:52 PM
Sounds good , In the mean time I would suggest familiarize yourself with this thread , 10-12bit thread and of course the  iso-research (https://bitbucket.org/hudson/magic-lantern/src/bd8e89d9dccf2d1541bb449efd06bb450cdce153/?at=iso-research) source code. (That's where I started from)
You will need a developing environment -- I use a VM (VirtualBox) on Window7 Pro with a preconfigured magic lantern clone of Ubuntu (http://www.magiclantern.fm/forum/index.php?topic=18259.msg174934#msg174934) ready to compile ML . Download links for clone of Ubuntu  here (http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989)
You can run the VM under Mac or Windows just get the correct version for your OS , if you feel adventurous you can setup your own environment @dfort has make some tutorial here (http://www.magiclantern.fm/forum/index.php?topic=16012.msg155422#msg155422) and here (http://www.magiclantern.fm/forum/index.php?topic=15894.msg154435#msg154435)
You need to be very comfortable with compiling , merging , update source with "HG" etc. ..... I work with a VM because it's faster & easier to get up and running in less then 30 mins .

Thanks for the recommendations, hopefully will make me helpful a bit faster.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: obiyan19 on July 18, 2017, 01:01:54 AM
for the 5d2 users could you do 2 screenshot , 1 with the latest nightly and the other with the adtg_gui

just because i read this tpic with interest, done with my 5d2 an 24-105 mounted on (& CF sandisk extreme 32 GB 120 mb/s udma 7)

the first image done with nightly

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2Fj7sgEa%2Fscreenshot_ML5_D2_nightly_18_07_2017.jpg&hash=335d61c8786370d061c76fd6a039c527) (http://ibb.co/j7sgEa)


and the second with adtg_gui.

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2FcwOSnv%2Fscreenshot_ML5_D2_adtg_gui.jpg&hash=93bef31a156a0498d24decc31d088be3) (http://ibb.co/cwOSnv)

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on July 18, 2017, 06:10:36 AM
Thanks @obiyan19 for the screenshots
nightly_18_07_2017
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fimage.ibb.co%2Fha68AF%2Fscreenshot_ML5_D2_nightly_18_07_2017.jpg&hash=213383d67d8e1e294592aaeef8db69bd) (http://imgbb.com/)

adtg_gui_2017_Apr04
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fimage.ibb.co%2FdQJicv%2Fscreenshot_ML5_D2_adtg_gui.jpg&hash=eb921533688cb20e7f534b2bae9bd631) (http://imgbb.com/)

Screenshots from @honza123 , thanks
Nightly_2017_Jul03
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fimage.ibb.co%2FdUxZHv%2FVRAM_ML_Nightly_2017_Jul03_5_D2212.jpg&hash=07eeab27f70fb5a9c3bcb1de9942bf13) (http://imgbb.com/)

adtg_gui_2017_Apr04
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fimage.ibb.co%2FiAaX4a%2FVRAM_adtg_gui_2017_Apr04_5_D2212.jpg&hash=9c6b4e5fc0c3b6bcd2bfb271b6ea9f38) (http://imgbb.com/)

Good thanks , this lines up with my 5d2 --I wanted to see the buffers from different 5d2 cams and to make sure I didn't corrupt
my ram , all good now to processed to the next level .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ErwinH on July 18, 2017, 03:15:46 PM
Here is a screenshot of the 550D running Nightly.2017Jul16.550D109 unified.
 
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2FkhneHv%2F550d.png&hash=ac9cc0aa3aa2c5b500ab1704ac7c4061) (http://ibb.co/khneHv)



Hope it helps.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on July 19, 2017, 07:37:30 AM
Thanks , yes all info is useful  ;D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on July 20, 2017, 08:28:57 PM
Hello, I joined just to post this encouragement for you to pursue this for 550D...
I'd LOVE to be able to use my 550D as a b-camera for filming 10 bit 24P raw 1080p!!!

Please be assured that your efforts will be appreciated and will keep many good cameras working for years to come!

I'm starting a new film this fall, and this would be excellent!

Please make it happen!
Thanks.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on July 28, 2017, 08:58:48 AM
I can now replicate the 3.5k resolution on my 5D2 but it's extremely fragile (I suspect the buffer I'm using & or how it's coded for liveview is little unstable)

 (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4310%2F36173282776_15a884200a.jpg&hash=e2b5db25853509b1793be0347291835b) (https://flic.kr/p/X7vwHf)
5D2 3.5k eXperimental  (https://flic.kr/p/X7vwHf) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4295%2F35409866793_ec5be132c1.jpg&hash=1227952dbc5ef841681be2f2a52f8a19) (https://flic.kr/p/VX3PJH)
LV-002_Liveview Raw Buffer Dump 5D2 3.5k eXperimental  (https://flic.kr/p/VX3PJH) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4305%2F35409898703_0754f41c19.jpg&hash=3088f5bec4a32c3437d6c0d0b2832f15) (https://flic.kr/p/VX3ZdT)
Screen Shot 5D2 3.5k eXperimental with ML B/W Preview (https://flic.kr/p/VX3ZdT) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4330%2F36079669101_f28ce32216.jpg&hash=8b52c7a7d3ee8bbb5ac3666851491bfa) (https://flic.kr/p/WYeJCn)
5D2 3.5k eXperimental Memory Patches (https://flic.kr/p/WYeJCn) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4294%2F35409821883_a834515a3a.jpg&hash=6d850bc8a72586a1d0fed57d37c3add2) (https://flic.kr/p/VX3Aop)
5D2 3.5k eXperimental Memory Patches Error (https://flic.kr/p/VX3Aop) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4313%2F35379315564_21629218d4.jpg&hash=2ab5110d2f8ec371984e7cddcdb0efb4) (https://flic.kr/p/VUmeVo)
5D2 3.5k eXperimental Free Memory (https://flic.kr/p/VUmeVo) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4325%2F36214860345_3a08406c80.jpg&hash=215621cd62a8daa32d22dcfac1a3dc40) (https://flic.kr/p/XbbCgT)
5D2 3.5k eXperimental Edmac ScreenShot (https://flic.kr/p/XbbCgT) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

I uploaded my adtg.log file to my google drive for those who are following along , I still need to analyzes the information for more developing but feel free to download it .
https://drive.google.com/file/d/0BxNY4d4Aa7KuQUNtM2V4UnY1UEU/view?usp=sharing

Still lots to understand as everyone can see , I could not change resolution or aspect ratio seem to be lock to 3584x1068 .
That could be because of the preset from the 3K 5d3 1:1 in the adtg_gui.mo with ISO research branch --not sure thou.
So to recap I loaded adtg_gui.mo , crop_rec.mo , raw_rec.mo(not mlv_lite) on  change set bd8e89d9dccf+ (iso-research) that I complied locally on my VM
1st: load 1:1 3x crop_rec setting
2nd: with the adtg_gui.mo enabled I activated the Digic register then the 3k 5d3 1:1 preset
then adjust the image with the cmos 1 & 2 .

I modified  raw.c , edmac-memcpy.c
in raw.c I add this
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
also in raw.c  changed to 0xC0F04500 from  0xC0F04508
Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04500
#endif
in edmac-memcpy.c change the r/w channel to 0xff & 0x0 from read 0x19 & write 0x03
Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
uint32_t edmac_read_chan = 0xff;
uint32_t edmac_write_chan = 0x0;

From the Liveview dump I can see that there is something there , seem to have the right size on the screen
Of course more reverse engineering needed , but there is some progress .  :)
 

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: nikfreak on July 28, 2017, 12:59:59 PM
May I just chime in and sum it up?


So frozen preview or not. i would say 1st goal would be to replicate what Greg/a1ex did AND be carefully with whatever you are trying out above this post  :o
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on July 29, 2017, 07:10:26 AM
Thanks for the links , gives me some ideas  ;)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: tonij on July 30, 2017, 02:48:55 AM
Hello, I just want to thank all the developers for the work done, I wish this will one day be ported to 7D, old camera but still doing an amazing job when used with ML.

+2

I don't post and there are probably many others like me but I find myself following along discussions like these.
All I see is page after page of green matrix symbols and sit here in awe at the ones who are able to stop bullets mid flight  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on August 23, 2017, 03:16:31 PM
@reddeercity

Just wanted to bump this because it looks like it was progressing nicely and suddenly stopped. Believe me I know how frustrating it is trying to get new features working. A while back I was trying to get lossless compression on the EOSM and 700D and had to put it aside. A while later ErwinH picked it up on the 700D and figured out the missing pieces. Following his lead I was able to get the EOSM working.

What made this possible was sharing the code on bitbucket. Checking up on your bitbucket account I see that you forked the ML repository a few times, the last activity was back in March. A lot has happened since then. If you could keep your development current in bitbucket maybe some of us could offer suggestions.

There are several good articles for working with bitbucket and Mercurial (hg) and if you have any questions there are ML forum members willing to help out.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on August 25, 2017, 10:00:54 PM
If you could keep your development current in bitbucket maybe some of us could offer suggestions.
+1
Please do that, I'm wanting to test out what you're doing, but the verbal 'patches' are just too cumbersome to keep up with.

Also what exactly is a 'liveview buffer'? How is it different from just a piece of memory?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: tecgen on August 29, 2017, 02:44:50 PM
+1

Could you please share the source code? Maybe you could simply push all your changes as one commit. We can compare the sources against the master/trunk within BitBucket afterwards.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 03, 2017, 02:06:21 AM
Sorry guys I haven't had time to responded to the posts , had one of my main servers crashed (lost the raid & the backup  :( ) so I had to rebuild from scratch which took some time.
 @dfort ,Ilia3101, tecgen
I really don't know how share my code on bitbucket , I work locally in my VM of Ubuntu which is very comfortable for me to develop in.
I tried once to set it up but could not figure it out , so I just kept working locally .
I did post my source code in my Bitbucket downloads , but all my changes are very simple I posted all my code changes and I thought I explained it in other posts.
I know this is not the way every one works , but for me it's the easiest way , so unless some one can tell me in very simple terms how you use/share my code on bitbucket correctly
with out  pointing to badly written tutorial I would gladly comply. This may slow down development , until then sorry I'll have to keep what I'm doing .

I'll try to explain what I have done and why . Here the Source Code (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magic-lantern_compressed_raw.zip) I been working from , it's has not been modified with my code .

I guess I should post a declaimer first -- I'm not a full time coder/hacker I know how thing work , I read code and understand perfectly just don't know the proper syntax when writing c code
I have to see example and do a lot of reading to make sure I'm doing the syntax correctly.


For this to work there needs to be a customs raw lv buffer , other then the canon which is for the default 1:1 (1880x1250)
So I followed a  tip (https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-125)  from a1ex about the 1100D in the 4K branch and I use the largest buffer I could find in the Edmac debug menu
I use the 3x crop mode buffer because it was big , I use this (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183023#msg183023) from Greg to help get the buffer but it unstable , could be because I did not disable the Canon default liveview buffer when I tested my code
In RAW.C  I added the new LV buffer
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
With out the buffer it will not work
The custom buffer is required, as ML redirects it only while recording.
In the crop_rec_4k branch I've used a SRM buffer (which can accommodate a full-res 14-bit picture).
This needs to be passed to SetEDmac as the first argument (instead of Canon's default buffer).
The buffer redirection without CONFIG_EDMAC_RAW_SLURP is fragile: it relies on lucky timing. It's best refactored somehow, but cache patching is also ugly...
Also in RAW.C I change the memory addressed for Liveveiw default was 0xC0f04508 to 0xC0f04500
Reason was it's the Photo Liveview buffer address and I thought logically it should have what I need to get to 3.5k and I was right to a certain degree .
Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04500
#endif
In edmac-memcpy.c I also changed the default R/W channels , default Read 0x19 & Write 0x03
Reason , it seem to been the r/w for the larger buffer I'm using , but this my not be the case as I could can not save a dng in the liveview buffer dump just the YUV 422 dumps
as per post #90 http://www.magiclantern.fm/forum/index.php?topic=19336.msg187766#msg187766
Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
uint32_t edmac_read_chan = 0xff;
uint32_t edmac_write_chan = 0x0;

That's It for code modifications the rest is done with  adtg_gui.mo with ISO research branch , by enabling 5D3 1:1 3k preset and adjusting the cmos registers .
See post #90 for details , not sure if there a better way to implement the custom buffer "RAW_LV_BUFFER_ALLOC_SIZE " this need more work to understand
I would say this would be the first thing to focus on -- that about it for now if you or any one have question about what I have done just ask I do my best to answer .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 03, 2017, 11:50:10 PM
Sorry guys I haven't had time to responded to the posts , had one of my main servers crashed (lost the raid & the backup  :( ) so I had to rebuild from scratch which took some time.
Oh no :(  Hope you didn't lose any important footage.

I know how thing work , I read code and understand perfectly just don't know the proper syntax when writing c code
Exact opposite here, know C, but have no idea how Magic Lantern or Canon works :D

I really don't know how share my code on bitbucket
Ok, well even in that case, you can still just upload a zip file with your modified code.
I think that is still better than describing what code you added to what files, as I don't know what part/line of the file you add stuff to, as that could matter in some cases, if you uploaded it I wouldn't have to worry about that as much.

But thanks, I will have another go and see if the same thing works on my camera tomorrow.

When you have the ~3500x1000px buffer showing in the menu, is the camera in a state where it can record a small test clip at that resolution?
Or would it just be noise and garbage? If so, what would the steps to getting a clean image be from that point onwards?

Also what do developers like a1ex think of reddeercity's work?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 05, 2017, 07:33:19 AM
@ Ilia3101 here you go Source Code with modification magic-lantern-adtg-gui-4k-test_source_code.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magic-lantern-adtg-gui-4k-test_source_code.zip)
In  edmac-memcpy.c Line 16 & 17 modified
In raw.c Line 78 , 116-118
in raw_rec.mo line 94 & 95 add resolution frame size 3840 & 4096 .
"Note" this is only working with raw_rec.mo not mlv_rec.mo
Exact opposite here, know C, but have no idea how Magic Lantern or Canon works :D
That good to know , that will help a lot  :D

When you have the ~3500x1000px buffer showing in the menu, is the camera in a state where it can record a small test clip at that resolution?
Or would it just be noise and garbage? If so, what would the steps to getting a clean image be from that point onwards?
Can't save a dng yet , tried a image dump I can save the liveview422 see  post #90 (http://www.magiclantern.fm/forum/index.php?topic=19336.msg187766#msg187766)  here is the original LV dump 422 image  LV-002.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-002.422) in Black & White ML perview  I guess I should correct my self it did save a DNG file but it was "0" bytes and there was a HD Liveview dump image HD-002.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/HD-002.422) which was normal looking .
I think to clean it up it could be as easy as fps A & B timing  or even the  Cmos  register , once there a clean LV image with ML B/W preview I think it may save a dng dump , there a raw record error when I try to save .raw file (Yes I'm still using raw 1st version not mlv lite , it's very basic just record raw once I have something stable I'll move to MLV Lite)

Also what do developers like a1ex think of reddeercity's work?
http://www.magiclantern.fm/forum/index.php?topic=19336.msg187776#msg187776
http://www.magiclantern.fm/forum/index.php?topic=19336.msg184661#msg184661
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183023#msg183023
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183200#msg183200
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183374#msg183374
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183461#msg183461
I guess I could go on & on , if you are asking is this a realistic development or even possible  , my belief is it is possible.
Or else I don't think I would have got so many tips from a1ex and nikfreak and other developer .

 @ Ilia3101 I will also PM you with the test 3.5k build I was using to do my test  that I posted in #90 , I don't what to post it publicly so it get out in the wild.
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 05, 2017, 05:12:34 PM
@ Ilia3101 here you go Source Code with modification magic-lantern-adtg-gui-4k-test_source_code.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magic-lantern-adtg-gui-4k-test_source_code.zip)
Thanks a lot, I'll see if I can come up with any useful changes.

Can't save a dng yet , tried a image dump I can save the liveview422 see  post #90 (http://www.magiclantern.fm/forum/index.php?topic=19336.msg187766#msg187766)  here is the original LV dump 422 image  LV-002.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-002.422) in Black & White ML perview  I guess I should correct my self it did save a DNG file but it was "0" bytes and there was a HD Liveview dump image HD-002.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/HD-002.422) which was normal looking .
I think to clean it up it could be as easy as fps A & B timing  or even the  Cmos  register , once there a clean LV image with ML B/W preview I think it may save a dng dump , there a raw record error when I try to save .raw file (Yes I'm still using raw 1st version not mlv lite , it's very basic just record raw once I have something stable I'll move to MLV Lite)
I have an idea why the preview is all noise: The 3K preset adjusts the preview parameters as well as raw parameters, so image framing is correct on 5D3 at the 3K res(is this true?), so maybe it ruins the preview on 5D2.
I'll try and find where the presets are in the code and comment out the changes in it that are irrelevant, maybe the image will be clear then.

@ Ilia3101 I will also PM you with the test 3.5k build I was using to do my test  that I posted in #90 , I don't what to post it publicly so it get out in the wild.
Thx
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 06, 2017, 07:53:04 AM
A small correction I did not use the 3x crop buffer in raw.c
Code: [Select]
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267) it's the 1:1 buffer , just look at post#90 in the Edmac screen shot on EDM#5 , address "43e28a4" 3570x1267 (3570/14*8 = 2040) . You could try the 3x crop buffer which is 4046 x 1127 so (4046/14*8 = 2312)
Quote
I have an idea why the preview is all noise: The 3K preset adjusts the preview parameters as well as raw parameters, so image framing is correct on 5D3 at the 3K res(is this true?), so maybe it ruins the preview on 5D2.
I'll try and find where the presets are in the code and comment out the changes in it that are irrelevant, maybe the image will be clear then.
Look at this for help adtg_gui.c-1127 (https://bitbucket.org/hudson/magic-lantern/src/bd8e89d9dccf2d1541bb449efd06bb450cdce153/modules/adtg_gui/adtg_gui.c?at=iso-research&fileviewer=file-view-default#adtg_gui.c-1127) and also this adtg_gui.c-1343 (https://bitbucket.org/hudson/magic-lantern/src/bd8e89d9dccf2d1541bb449efd06bb450cdce153/modules/adtg_gui/adtg_gui.c?at=iso-research&fileviewer=file-view-default#adtg_gui.c-1343)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 07, 2017, 05:13:57 AM
Found some helpful information for "CONFIG_EDMAC_RAW_SLURP"  treatment on 5D2 , as many times I read over this code I never see this  :o
I think this is a better way then "RAW_LV_BUFFER_ALLOC_SIZE" but not sure if this works on digic 4 cam's  so far it works with digic5 ,
I'll give it try , this is the link to the code starts at line 407 allocate-raw-lv-buffer#Lsrc/raw.cT407 (https://bitbucket.org/hudson/magic-lantern/commits/1a2a91eb31a38ad0a3b118f14b64229e8823ecc1?at=allocate-raw-lv-buffer#Lsrc/raw.cT407)
Code: [Select]
408  * How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP:
409  * - Go to LV and use lv_save_raw
410  * - Check the RAW_LV_EDMAC debug info
411  *  - Suppose it reports W: 0xA3A H: 0x3C7 (taken from 1100D in LV mode)
412  * - EDMAC W is the number of bytes per line
413  *   - The W resolutions is computed as: W * 8 / 14 (raw buffer is 14 bits per pixel)
414  *   - Thus 0xA3A  8 / 14 -> 1496 pixels
415  * - EDMAC H is the number of "jumps"
416  *   - The H resolutions is H + 1 -> 0x3C8 -> 968 pixels

Edit: This will have to be add to "platform / 5D2.212 /  internals.h"
Code: [Select]
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP
not to sure about "connection #0" thou -- this maybe a wild goose chase
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 07, 2017, 09:45:04 AM
Haven't really managed to get anywhere ::)
I tried changing the presets, separating the 3k one out in to 2 parts, but didn't really help.

I suspect I might have the same exact 'soft brick' you complained about before, as I had tried out your changes a while ago and seen results, now I'm not seeing anything.

Every time I try and do anything with magic lantern I always slow down eventually and give up for a bit, then I try again and the cycle repeats :(

Interesting about the SLURP thing though.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 07, 2017, 09:53:11 AM
not to sure about "connection #0" thou -- this maybe a wild goose chase

Connections are probably hardcoded (or configured in a way we don't currently understand) to various image processing modules. So far, all the D4 and D5 models get the LiveView RAW data from connection 0. Connection 6 and 7 are pass-through (whatever you transfer there using a Read EDMAC channel will be copied on the other side, to a Write EDMAC channel configured for the same connection).

See this diagram (from http://www.magiclantern.fm/forum/index.php?topic=18315.msg188630#msg188630 )

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fcapture%2F5D3-playback.gif&hash=edab14fc4cc5acea3d2768d1d111e5fa)

A read EDMAC channel will read the data from RAM and send it to some image processing module.

A write EDMAC channel will get the data from some image processing module and will write it into RAM.

The data can be read via some input module (such as DSUNPACK, ADUNPACK, UNPACK24, or others - possibly unnamed), where you can configure the input bit depth. In this case, the input stream can be 10-bit, 12-bit, 14-bit or 16-bit, configured using DSUNPACK_MODE / ADUNPACK_MODE / UNPACK24_MODE / 0xC0F371FC (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620) / etc. In this case, the image processing module that actually does the work (e.g. JPCORE) probably receives normalized data.

A similar process happens on the output side, where a PACK module is used (PACK16, PACK32). Remember the PACK32_ISEL and PACK32_MODE (http://www.magiclantern.fm/forum/index.php?topic=5601.msg174177#msg174177):
- PACK32_ISEL probably means "wire the input of PACK32 module to whatever other image processing module that outputs Bayer data, in various places in the pipeline (http://www.magiclantern.fm/forum/index.php?topic=18393.0)";
- PACK32_MODE configures the output bit depth of whatever image data arrives to the PACK32 module (10/12/14/16).

Currently, the uncompressed bit depth selection is done in raw_lv_request_bpp (raw.c, crop_rec_4k branch).

Experiments on the above can be made on existing code that's known to work (raw_twk for digic 4/5, EekoAddRawPath for digic 5), or on FA_MaxSelectTestImage / FA_SubtractTestImage (low-hanging fruit for understanding the image processing modules).

One interesting note from the crop_rec_4k thread, where I've implemented the 10/12-bit lossless compression by darkening the input raw data (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183106#msg183106) (so the input and output would still be 14-bit, but there will be fewer levels actually used by the image - as many as a 12-bit or a 10-bit image). For lossless compression, the entropy of a 10/12-bit stream is similar (maybe identical?) to the entropy of a 14-bit stream with each value shifted by 4 or 2 bytes (integer division by 16 or by 4).

How does that work?

raw.c:raw_lv_request_digital_gain:
- lv_raw_gain is written to SHAD_GAIN_REGISTER
- RAW_TYPE_REGISTER is set to 0x12 (DEFCORRE) (http://www.magiclantern.fm/forum/index.php?topic=18393.0)
- this image happens to be scaled by digital ISO gain and is not affected by bad pixels.

When the digital gain is not set, RAW_TYPE_REGISTER is set to CCD (probably the first stage where the raw data gets in the digital domain).

For a better understanding, set RAW_TYPE_REGISTER to DEFCORRE (0x4 on digic 4) without overriding SHAD_GAIN, and notice what happens at ISO 320 vs 400. Repeat for RAW_TYPE_REGISTER set to CCD. Then start overriding SHAD_GAIN with any values you want, even something like this:
Code: [Select]
if (get_halfshutter_pressed())
{
    EngDrvOut(SHAD_GAIN_REGISTER, rand());
}


Now look at the problems that appeared from this change (the 10/12-bit lossless implementation):

- First of all, the "slurp" EDMAC channel (the one that writes the raw data into memory) had to be configured for exact resolution; the autodetection from raw_lv_get_resolution (0xC0F0680x/0xC0F0608x) is not exact - it's often off by one on the vertical direction, although the exact reason is unknown. Being off by one on 5D3 resulted in the raw data being correct only on every other frame (although I don't really understand why that happens). The issue was only with RAW_TYPE_REGISTER set to DEFCORRE, but it all worked fine when it was set to CCD.

- Next, take a look at this bug (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183246#msg183246): in a video mode with increased vertical resolution, the darkening works only on the top side, on an area equal to default Canon resolution in that video mode. That means, we have to reconfigure some more registers (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183181#msg183181) - probably in the image processing pipeline, all the way from CCD (sic) to DEFCORRE. Which ones? I don't know - couldn't find them in adtg_gui. I hope to find them by emulating the LiveView in QEMU, but that's going to be a really long journey.

That's why, for now, the 10/12-bit lossless compression only works in video modes with unmodified resolution (plain 1080p, plain 720p and 5x zoom).
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 08, 2017, 05:51:43 AM
@Ilia3101 -- baby steps  ;D just pick one thing to focus on and learn as much as you can , look closer at what the code is doing in the 3k 5d3 preset , there a lot of clues there even thou it for the 5d3 the 5d2 share some similarities in this respect
it may not look like there much progress right now but we are laying out the frame work for more then 3.5k video this will also lead to FHD 10/12bit and even in time compressed raw .

@a1ex thanks for the lesson , things are getting clearer .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Wayne H on September 08, 2017, 07:18:20 AM
@Alex that is an excellent and highly detailed explanation on how you Magic Lantern magicians are squeezing 3.5K and 4K out of the 5d3,

I was wondering a while back why the 12bit and 11-8bit lossless options were only available on the 3.5K mode, and not in the other UHD and 4K modes, but after doing some magic lantern reading, i quickly understood that the 3.5K mode works differently to all the other high resolution modes,

Makes me appreciate just how much work you guys do in your own free time, LIVEVIEW is incredibly complex. (it took Canon 6 months just to get the 5d3 to spit out clean and mirrored HDMI via a firmware update) AMAZING WORK GUYS.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 03, 2017, 07:23:33 AM
Ok I'm going to try and get "CONFIG_EDMAC_RAW_SLURP" for 5D2 working ,  from this allocate-raw-lv-buffer#Lsrc/raw.cT407 (https://bitbucket.org/hudson/magic-lantern/commits/1a2a91eb31a38ad0a3b118f14b64229e8823ecc1?at=allocate-raw-lv-buffer#Lsrc/raw.cT407) tell how to go about it , but I still don't understand .
Code: [Select]
 
408  * How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP:
409  *   - Go to LV and use lv_save_raw
410  *   - Check the RAW_LV_EDMAC debug info
411  *   - Suppose it reports W: 0xA3A H: 0x3C7 (taken from 1100D in LV mode)
412  *   - EDMAC W is the number of bytes per line
413  *   - The W resolutions is computed as: W * 8 / 14 (raw buffer is 14 bits per pixel)
414  *   - Thus 0xA3A  8 / 14 -> 1496 pixels
415  *   - EDMAC H is the number of "jumps"
416  *   - The H resolutions is H + 1 -> 0x3C8 -> 968 pixels

from line 411 -- W: 0xA3A H: 0x3C7  how do I figure out what these compute to , I must be missing something here . Also how do I implement "lv_save_raw" from line "409" there must be some code I missed . I remember some where it was posted but not sure where , @dfort didn't you go though this with the eosm ? I can't seem to find it , if not I have to do more searching .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on October 05, 2017, 01:43:07 AM
@dfort didn't you go though this with the eosm ? I can't seem to find it , if not I have to do more searching .

I can't find it either. This is the closest that I could come up with: http://www.magiclantern.fm/forum/index.php?topic=5601.msg179673#msg179673

Maybe write something in dubug.c calling lv_save_raw and print it on the screen or console? There's a hint in lv_rec.c:

Code: [Select]
    /* this causes the function to hang!? */
    if(data.options.rawMode)
    {
        //~ bmp_printf( FONT(FONT_MED, COLOR_WHITE, COLOR_BLACK), 30, 20 * yPos++, "Make sure you ran call('lv_save_raw')");
        call("lv_save_raw", 1);
        msleep(200);
        raw_update_params();
    }

The rest of it makes sense to me:

0xA3A (hex) = 2618 (decimal)
2618 *  8 / 14 -> 1496 pixels
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 05, 2017, 06:21:21 AM
I can't find it either. This is the closest that I could come up with: http://www.magiclantern.fm/forum/index.php?topic=5601.msg179673#msg179673

Maybe write something in dubug.c calling lv_save_raw and print it on the screen or console? There's a hint in lv_rec.c:

Code: [Select]
    /* this causes the function to hang!? */
    if(data.options.rawMode)
    {
        //~ bmp_printf( FONT(FONT_MED, COLOR_WHITE, COLOR_BLACK), 30, 20 * yPos++, "Make sure you ran call('lv_save_raw')");
        call("lv_save_raw", 1);
        msleep(200);
        raw_update_params();
    }
@dfort thanks , that will help a lot


The rest of it makes sense to me:

0xA3A (hex) = 2618 (decimal)
2618 *  8 / 14 -> 1496 pixels
:-[ I never thought hex , I light bulb just came on as soon as I that.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on October 05, 2017, 07:50:01 AM
It might help if you read how dmilligan found the addresses for the EOSM starting around here:

https://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969

Yeah, it is confusing and doesn't translate across platforms but it might get you a step closer.

There's also several methods of finding some of this stuff. It appears that in the code snippet you posted it is showing, "How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP" using the 1100D as an example. The size is 1496 x 968. Isn't that the size of the full raw buffer?

Another way of getting the full raw buffer is to set the camera's movie settings to the mode you want (say 1920x1080/24 in the Canon menu) and while the camera is in movie mode shoot a simple silent still DNG. Take that DNG and run it through dcraw using the -4 and -E options. The output file (in .pgm format) will be the size of the full raw buffer. That's what I had to do when working out the focus pixel maps for cameras that show focus pixels in raw video.

To get the entire raw buffer from a DNG, including OB area: dcraw -4 -E
To get the active area only: dcraw -4 -D

That's the second time today that I quoted this tip. OB means out of bounds. There's actually an area that gets cropped out when viewing the DNG of the full raw buffer and using that -E option will show you the entire raw buffer.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 30, 2017, 04:00:53 AM
@dfort thanks , I'll give that a go
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 30, 2017, 05:34:24 AM
So some more tests -- with these latest attempt I  see some progress in  Liveview  @  3584x1068  (http://www.magiclantern.fm/forum/index.php?topic=19336.msg187766#msg187766)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fpreview.ibb.co%2Fmjngfm%2FVRAM4.png&hash=763f19b03c0e26edfcd68657dda86081) (http://ibb.co/m7vZ0m) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fpreview.ibb.co%2FjVihmR%2FVRAM5.png&hash=ce9740ac50df7b47e615d9e22e7e395f) (http://ibb.co/c2Brfm)

The first image with the pink noise bar across the top of liveview was with a half shutter press and the second was with Zebras & Focus Peck enable -- see image below

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fpreview.ibb.co%2Fg8ef6R%2FVRAM6.png&hash=180d864d342c7e848538b673566236e0) (http://ibb.co/ifEDRR)

So this leading me to believe the there is either a vertical resolution out of sync or fps timers .
I guess I should say what my configuration was  ;D I'm still using my custom build with modified Liveview buffer  (http://www.magiclantern.fm/forum/index.php?topic=19336.msg189550#msg189550) as noted in other post here on this thread .
Camera setting are real basic , Preview I use "ML B/W preview for liveview , frame override enable to 23.976 . I only loaded adtg_gui.mo , crop_rec.mo , raw_rec.mo (I use .raw format for testing)

I notice something very interesting , on first boot I enable adtg_gui.mo & the rest -- mess with  cmos regs etc. ... when I got the image all messed up I did a reboot to clear cmos registers and that's when it happen  :o it automatically boot to my custom buffer , now I was back to 3584x1068 -- the strange thing adtg_gui.mo was not enable just crop_rec 1:1 , there was no memory patches . That what the screen shots are from , the non enabled cmos registers , just crop record . The code from crop_record comes from the 5D3 and I have not change it at all , I should mention before I rebooted and adtg_gui.mo enabled I had use the 5d3 3K preset so unless I rebooted to quickly and those cmos registers so how stayed in memory I can't seeing it but you never know. So as far as I know all this is done with the custom liveview buffer I'm using
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
Plus the other changes I made to the code here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg189550#msg189550)
Setting:
crop_rec.cfg
Code: [Select]
# Config file for module crop_rec (CROP_REC.MO)crop.preset = 1magic.cfg
Code: [Select]
# Magic Lantern Nightly.2017Apr14.5D2212 (bd8e89d9dccf+ (iso-research))
# Built on 2017-04-14 03:42:23 UTC by ml@ml-pc
# Configuration saved on 2017/10/29 19:17:40
beta.warn = 29
menu.first = -3
movie.log = 1
rec.notify = 0
enable-liveview = 2
fps.override.idx = 11
battery.drain.rate.rev = 69
hist.log = 0
spotmeter.draw = 0
zebra.draw = 0
disp.mode.x = 149

raw_rec.cfg
Code: [Select]
# Config file for module raw_rec (RAW_REC.MO)
raw.video.enabled = 1
raw.res.x = 20
raw.dolly = 1
raw.preview = 2
raw.warm.up = 2

I also tested this with HDMI output connected my atomos ninja hdmi hard drive recorder , uploaded to my bitbucket downloads link below
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/00150_H264.mp4
there's some shearing etc. ... but did not crash the camera , one thing I notice thou was the ml scopes where blank no feed back for the image .
at the end of the video I try to record raw video but got a "Raw error" this was expected .
But in the next file after I cycled the hdmi connection I then had the ML scopes came back see it in the link below
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/00153_H264.mp4

So I can also take a CR2 photo in this configuration and it all right no problem below is a short video that I take with my iPhone with me pressing the shutter fully to get a cr2 photo link below also
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/IMG_2273.MOV
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8668.CR2

I notice as I panned the camera left to right the preview that was ml b/w the distortion would move or pulse and of course with half shutter I get color distortion , but it's not a static frozen image







Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 30, 2017, 06:04:18 AM
I notice there one line of pixel you can see here the LV-004.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-004.422) image
Liveview dump
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fpreview.ibb.co%2FmQPYt6%2FLV_004.jpg&hash=2ce06c45e2bcc8de8b23c69a03d50ac5) (http://ibb.co/ktpzY6)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 07, 2017, 06:11:25 AM
Going over ARM Configuration Bus Access to Peripherals (http://www.ti.com/lit/ds/symlink/tms320dm368.pdf) which I been led to believe to be in the 5d2 and all digic 4

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2Fmvdusb%2FArms_1.jpg&hash=95fe034ce2d3ecc07fd94b725422e5dd) (http://ibb.co/mvdusb)

Look at the red hi-lighted box , that looks like the resizer registers I have been adjusting in my earlier tests I posted with adtg_gui.mo 
the IPIPE Registers are interesting DM365 IPIPE register set (http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/100/t/36802)
Quote
....These include for example register offset ranges 0x44-0x1CC and 0x1F0-0x228.
According to the code these address ranges contain registers for OTF Defect Pixel Correction, Lens Shading Correction, Noise Filtering and CFA interpolation. ....
will that seem to be dslr stuff  :o  , need to dig deeper .


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 07, 2017, 06:27:02 AM
Could this be the Sensor Interface (ISIF) – RAW Data Processing Flow for canon digic 4 or all eos ?

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2FnfKg5w%2Fcmos2raw_flow.jpg&hash=a1fe2203858d4438d9134e437d4da675) (http://ibb.co/nfKg5w)

http://www.ti.com/lit/ug/sprufg8c/sprufg8c.pdf

I think this my be a very importance pdf with registers that could map out to what is needed for development


Edit: interesting , Image Pipe Interface with dark frame subtraction ?
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2FgqiVyG%2Fcmos2raw_flow_1.jpg&hash=ae37d1c0bee76ad73667fd6bf747f0c2) (http://ibb.co/gqiVyG)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on November 08, 2017, 10:23:15 AM
I have nothing valuable to add I just wanted you to know I get so excited when there's a new reply in this thread.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on November 09, 2017, 07:02:56 PM
I have nothing valuable to add I just wanted you to know I get so excited when there's a new reply in this thread.

+1  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on November 09, 2017, 07:22:33 PM
I have nothing valuable to add I just wanted you to know I get so excited when there's a new reply in this thread.

Agreed!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: domo94 on November 29, 2017, 09:37:52 PM
I am coming from a 550d thread.

To help with further development with 550d, and hopefully to get this software ported with Canon 7D to get the most consistent and solid 10bit 1080p recording I need!

How can I help?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on November 30, 2017, 07:12:14 AM
There are lots of parts to this.

There's the 10bit/12bit (http://www.magiclantern.fm/forum/index.php?topic=5601.0) part that apparently requires either CONFIG_EDMAC_RAW_SLURP and/or CONFIG_RAW_LIVEVIEW but that can't be right because if you notice that there is an experimental build for the 1100D that supports 10bit/12bit using a branch where RAW_SLURP isn't defined on that camera. However, if RAW_SLURP is required that has been cracked on the 1100D in the allocate-raw-lv-buffer branch.

Then there's lossless compression where it seems that we found all the stubs for the 7D (http://www.magiclantern.fm/forum/index.php?topic=18443.msg193534#msg193534) but haven't figured out how to get it working properly yet.

Finally, there's the crop_rec module (http://www.magiclantern.fm/forum/index.php?topic=17021.0). Several cameras are working with the 3x3 mode which allows shooting nice quality 60fps raw video without that terrible line skipping that shows lots of aliasing artifacts in the 1280x720p mode. However, to get to the juicy stuff that includes 4k raw video will require a lot more work that so far is only supported on the 5D3.

Attacking this all at once is overwhelming, believe me I tried. I'd suggest taking one of these topics and start reading from the beginning like a novel to see how these puzzles were eventually solved.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 01, 2017, 06:48:59 AM
Actually it's really ISO_Research Branch with crop_record that's it . Here (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magic-lantern-adtg-gui-4k-test_source_code.zip)  is my source code I'm working from 
If you read my thread from the start it's all explained.
10/12bit & lossless compression is not part of this right now so has no bearing on Digic IV UHD/3.5K raw video development ,
Thou I will implement 10bit (as this work on 5D2) after I can get a unscrambled stable image  (very close now)

Digic IV cams work with RAW_LV_BUFFER_ALLOC_SIZE  as long as you find the right buffer e.g. 5D2
Code: [Select]
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)I would suspect 550d would be the same but for 7D it may not because of the dual cpu there are more difficulty

I am coming from a 550d thread , To help with further development with 550d

Thanks , look here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183277#msg183277) Greg was helping with 500d with some promising result but stop.
So search Greg's posts on my thread & on the forum he has some very good info to share e.g. 500D, trying to increase the resolution of the video raw (crop mode).  (http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909)
In fact from his initial experiment's a1ex started the whole 4K/UHD for 5D3  crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView  (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182052#msg182052) he credited in the first post .

If you can compile the source code (you need a working environment I recommended a prebuild VM of Ubuntu  Quick Guide and Solutions for VirtualBox  (http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989) that the one I use)
can be ran on Windows or MAC , It's all setup nothing to be configured --install VirtualBox for your OS and start working.
The other useful thing is to decompile your ROM Dump files look here (http://www.magiclantern.fm/forum/index.php?topic=12177.msg117735#msg117735) for help in disassembling.

...... ported with Canon 7D to get the most consistent and solid 10bit 1080p recording I need!

This is off topic but here you go  ;)
Will there is 10bit 1080p already with the 7D with the experimental 10/12bit build , but only works in 3x cropmode ,
can be downloaded from my bitbucket account 10bit_12bit_raw_2016Dec01.7D203.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-10bit_12bit_raw_twk_crop_rec.2016Dec01.7D203.zip)
1:1 liveview produces corrupted image every other frame usually the top 1/3 , that's why it's not in the main build area and only works in cropmode with audio by the way  :)
 


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 01, 2017, 09:59:02 AM
Yes, I've been following this topic and I am able to compile Magic Lantern and disassemble ROMs. In fact I compiled that 7D build you pointed to on your last post exactly one year ago. I took it down because 10bit/12bit wasn't supported on the 7D (or 5D2, 50D, 500D, 550D).

According to the experiments download page (https://builds.magiclantern.fm/experiments.html) "Only models with CONFIG_EDMAC_RAW_SLURP are compiled." However, I thought it was worth noting that there is a 10bit/12bit build for the 1100D available for download. It is being compiled on the raw_video_10bit_12bit branch and if you look through the code you'll see that CONFIG_EDMAC_RAW_SLURP isn't defined for the 1100D in that branch.

Digic IV cams work with RAW_LV_BUFFER_ALLOC_SIZE  as long as you find the right buffer e.g. 5D2
Code: [Select]
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)

You defined RAW_LV_BUFFER_ALLOC_SIZE in raw.c but you didn't define CONFIG_EDMAC_RAW_SLURP in internals.h. You are working on the iso-research branch but for this experiment you will probably have better luck using the allocate-raw-lv-buffer branch. Note that neither iso-research or allocate-raw-lv-buffer has the 10bit/12bit code so you'll need to merge those changes with the raw_video_10bit_12bit branch. [EDIT - Yeah, you're doing all that.] For everything you're trying to do your goal should be to eventually get it working in the crop_rec_4k branch. Good luck!

Note: It would really help if you would update your Bitbucket repository (https://bitbucket.org/reddeercity/) instead of posting a link to your local repository.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 02, 2017, 08:11:15 AM
Let me backpedal a bit. This thing is rather complicated and hard to grasp all at once.

Instead of blabbing away let me offer a bit of code to find something that might help. A while back I was searching for the SRM_BUFFER_SIZE on some cameras that I didn't have access to so I made a custom build to print it out on the LCD screen. This is something that you will probably need as a1ex noted in this post (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183374#msg183374).

src/exmem.c  (based on the unified branch)
Code: [Select]
static void srm_malloc_cbr(void** dst_ptr, void* raw_buffer, uint32_t raw_buffer_size)
{
    if (!srm_buffer_size)
    {
        /* we can't tell how much to allocate; the allocator tells us */
        srm_buffer_size = raw_buffer_size;
    }
    else
    {
        /* it should tell us the same thing every time */
        ASSERT(srm_buffer_size == raw_buffer_size);
    }
   
    /* return the newly allocated buffer in the output variable */
    *dst_ptr = raw_buffer;
   
    /* announce it's done */
    give_semaphore(srm_alloc_sem);
   
    bmp_printf(FONT_LARGE, 50, 300, "SRM_BUFFER_SIZE 0x%x", srm_buffer_size);
}

The only change is that last line in order to print the SRM_BUFFER_SIZE on the screen when shooting a simple silent DNG, like this on the 7D:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4552%2F38744482622_fa7c746eb2.jpg&hash=90981a0cd931d75cdd81bdf041425227) (https://flic.kr/p/222HAR5)

So this can be added to the consts.h file (malloc section):

Code: [Select]
#define SRM_BUFFER_SIZE 0x1F80000   /* print it from srm_malloc_cbr */
@tecgen found it for the 550D (http://tecgen) - 0x1F68000 - I should add that he found it on his own.

He said he was going to try to find it on the 50D and 5D2 but I haven't seen any updates from on this. So how about we look for these? I put up builds for these cameras on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) so we can document the SRM_BUFFER_SIZE. If anyone wants to look for it on a camera that isn't listed I'd be glad to post a build or it can be compiled from the unified-find-SRM_BUFFER_SIZE branch (https://bitbucket.org/daniel_fort/magic-lantern/src/3418e399402f28ef7ab6930284a0d8f2fd39b0e3/?at=unified-find-SRM_BUFFER_SIZE) in my repository.

I also did a screenshot of edmac on the 7D and see something very different from the 5D2. Maybe the channels we want are running on the master processor so they aren't visible on the slave? I don't know.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4569%2F38747827772_fa536d6542.jpg&hash=d213ac5204cd16b89337aa11f6d74a82) (https://flic.kr/p/2231Kf5)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Andy600 on December 02, 2017, 10:38:37 AM
@dfort - Just ran your build on the 50D and the SRM buffer size shows as 0x1b14000

useful bit of code :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on December 02, 2017, 06:49:30 PM
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fimage.ibb.co%2Fnb2Tzb%2FVRAM0.png&hash=adc072e527fcb89b7afda06775a59f5c)

I can finally 'help' with getting 3k on the 5D2! thx dfort

Code: [Select]
0x25d0000
37MB buffer?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 04, 2017, 06:55:12 AM
From adtg_gui.c-1121 (https://bitbucket.org/hudson/magic-lantern/src/50d8f06018dc52f72f04265432dd83106e495e85/modules/adtg_gui/adtg_gui.c?fileviewer=file-view-default#adtg_gui.c-1121) This is the 3K 5d3 preset I use on my 5D2
Code: [Select]
static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6804:                /* C0F06804 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
with buffer
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
that end up to be 3584x1068 .
A hint from nikfreak from here (http://www.magiclantern.fm/forum/index.php?topic=19336.msg183107#msg183107) I see where I my have a problem , just dawn on me
that the registers are a little different 
•C0F06804 (Digic5)
•C0F06084 / C0F06088 (Digic4)

Dam Dyslexia  >:(

As seen in this piece of code adtg_gui.c-126 (https://bitbucket.org/hudson/magic-lantern/src/50d8f06018dc52f72f04265432dd83106e495e85/modules/adtg_gui/adtg_gui.c?fileviewer=file-view-default#adtg_gui.c-126) full res liveview uses C0F06084
Code: [Select]
   * 5D2 photo:
     * 0xC0F06088.hi - 0xC0F06084.hi is the vertical resolution (3804)
     * 0xC0F06084.hi shifts the image in one-line increments, so it must be the first scanned line
     * (0xC0F06088.lo - 0xC0F06084.lo) * 2 is the horizontal resolution (5792)

So it look like I need to make a few changes to my source code to take that in account .
Stay tuned ;)


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on December 04, 2017, 07:13:28 AM
This is all smoking hot... keep it up guys!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 04, 2017, 07:24:51 AM
Something extra to add , I thing I know way I'm lock to 3584x1068 , I see that cmos[1] is set to 0x802 and if
I convert the hex to decimal so 0x802 = 2050 *14 /8 = 3587.5 -- my custom buffer 2040x1267 , so I my change the buffer to 2050 instead of 2040 and see what happens
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on December 04, 2017, 05:27:54 PM
A very hot topic, indeed!  I keep checking for new posts at least twice a day.  Am a right by thinking that there might be some hope for the good old 7D too?  Excellent work, guys!  I keep my thumbs pressed!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 04, 2017, 06:55:41 PM
Checking it twice a day? Really?

Of course it would be much easier if we all upgraded our older cameras for something not so old but not too new either.

Anyway--I did some playing around with the 7D over the weekend. One of the things I wanted to see was all the changes that @reddeercity actually did so I found which changeset he started with and managed to put it into a branch in my repository. This (https://bitbucket.org/daniel_fort/magic-lantern/commits/deee398f9f00c674b791cee7d922e0327a571e6a) should provide a clearer picture of what is going on. I also made a dummy pull request of the stuff I worked on which was trying to get CONFIG_EDMAC_RAW_SLURP working. I used the allocate-raw-lv-buffer branch because it was pointed out somewhere to look at how the 1100D was done done on this branch but it didn't compile! Turns out that it still had a problem in the code that was resolved a while back. BTW, the 1100D is an EVF_STATE camera while the 50D, 5D2, 7D and other cameras we're looking at are LVState cameras. None of the LVState cameras have CONFIG_EDMAC_RAW_SLURP working--yet. Anyway, here's my work in progress:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff

There are lots of pieces to this puzzle and it would be great to get at least some of the answers we found cataloged in some way. David (a.k.a. reddeercity) how about modifying your first post and adding the SRM_BUFFER_SIZE for the various cameras that we have this resolved? At some point down the line we'll need this and having them in one place instead of scattered throughout various posts would be much easier to work with.

Another thing worth sharing are these screenshots I did on the 7D.

First the default:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4555%2F37936637285_b425e1546a.jpg&hash=ec0f65cca9629ded4c34fcf973020f35) (https://flic.kr/p/ZNkbzc)

Next, what happens after recording a bit of raw video again at the default raw video settings:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4556%2F38106830844_5c0fb2db6b.jpg&hash=eaa2249818b605d5bc3dc604da6329b3) (https://flic.kr/p/214ntbU)

Of significance is what happens with channels 6 and 10. Note this in the code:

edmac-memcpy.c
Code: [Select]
#elif defined(CONFIG_7D)
uint32_t edmac_read_chan = 0x0A;  /*Read 0x19 0x0D 0x0B 0x0A(82MB/S)*/
uint32_t edmac_write_chan = 0x06; /* Write 0x5 0x6 0x4 (LV) */

Channel 6 is the write channel and 10 (0x0A hex) is the read channel.

Now let's put the camera in zoom mode, max out the image size and record a clip--albeit a very short clip.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4580%2F37936638545_1952fba3c7.jpg&hash=724f70e24cab5ded7f49da0bded79070) (https://flic.kr/p/ZNkbWV)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4529%2F38106832204_245b7a9bda.jpg&hash=78ba5fb7401e0661584044422d987c24) (https://flic.kr/p/214ntAm)

Pretty cool, eh? (Speaking Canadian.)

Not sure what channel 16 is for but it is interesting that it is the largest and the address didn't change.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 05, 2017, 01:30:52 AM
 :) Yea , 2 more things -- wear a toque get a beer and I will make you a honorary Canadian  ;D
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fen%2Fthumb%2F2%2F28%2FBob_and_Doug_McKenzie.jpg%2F275px-Bob_and_Doug_McKenzie.jpg&hash=86259e25eef84cddd2d10dcb38a2d9cb)

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 05, 2017, 02:06:55 AM
Ok now down to business -- That large buffer is what I would focus on from channel #16 , That's how I choose the 5d2 buffer on channel #5 in the Edmac.
If you navigate  to channel #16 on the 7D I would expect to see a different address then the default in raw_rec.c
Code: [Select]
#if defined(CONFIG_500D) || defined(CONFIG_550D) || defined(CONFIG_7D)
#define RAW_LV_EDMAC 0xC0F26008
#endif
That's why I changed
Code: [Select]
define RAW_LV_EDMAC 0xC0F04508 to
Code: [Select]
define RAW_LV_EDMAC 0xC0F04500Something interesting -- if you take Edmac#16 (on 7D) buffer "4410x1225"
4410/14*8 = 2520
So this could be your custom buffer 2520 x 1127 1225-- this is how I came up with my buffer.

....... at least some of the answers we found cataloged in some way. David (a.k.a. reddeercity) how about modifying your first post and adding the SRM_BUFFER_SIZE for the various cameras that we have this resolved?

Yes I will update the first post with screen shots and buffers information on all digic iv that are being actively be worked on ,
this means any one following this thread with a digic iv cam that don't see your camera on the first post will need to be involved.
At this point the camera's with CF Cards have priority 5D2 , 7D , 50D -- then the 550d & 500d but not limited in that order 

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 05, 2017, 03:05:09 AM
Also I use the crop_rec.mo  to get the 3.5k . I enable the 1:1 preset which I sure give me every line etc. ...
Code: [Select]
  switch (crop_preset)
        {
            /* 1:1 (3x) */
            case CROP_PRESET_3X:
                /* start/stop scanning line, very large increments */
                /* note: these are two values, 6 bit each, trial and error */
                cmos_new[1] = (is_720p())
                    ? PACK12(14,10)     /* 720p,  almost centered */
                    : PACK12(11,11);    /* 1080p, almost centered */
               
                cmos_new[2] = 0x10E;    /* read every column, centered crop */
                cmos_new[6] = 0x170;    /* pink highlights without this */
                break;
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on December 05, 2017, 12:28:13 PM
Checking it twice a day? Really?

Of course it would be much easier if we all upgraded our older cameras for something not so old but not too new either.

O, absolutely!  Of course, you are right, Dfort, it would be much better if we upgraded our cameras to something newer but do we really have that choice?  The only alternative and better model to the 7D that supports Magic Lantern is the 5D3 but how many people on this forum can afford this body and the lenses for it?  I also own the 100D which is a fantastic little ML camera, perfectly suited for travel, especially after the latest ML developments, but it deffinitely is not appropriate for serious work, considering the short battery life, the limited vertical resolution, lack of VAF filter and especially the very annoying "hick-ups" (random jumps throughout the video) in the 5x-magnification mode which makes the camera unuseable in that most important for serious practical video shooting mode.

It may be interesting for you to know that this is not only my opinion.  I have discussed the issue with several  friends and colleagues of mine who use their 7Ds, 5D2s and even 50Ds on a daily bases and they all share these same thoughts.  That is why, the excitement about the latest developments concerning these old Digic IV cameras among the video shooting community is tremendous!  I am sure, Mr. Reddeercity who obviosly loves his 5DMk2 so much, understands what I mean :-))).

And finally, a moral issue that I would like to share here.  You say, upgrading to newer models. But why, if the old models continue working so well, are built like tanks, are extremely reliable and in many practical situations outperform the newer ones?  Why would I want to sell my trusted 7D that I payed so much money for back in 2011 if it still works and looks like brand new and in fact, I have not used up even 50% of its shutter life?  How much does it cost to recycle a 7D and how much energy would it take?  Wouldn't we all make a friendly gesture to our Mother Nature if we use our cameras more consciously and do not upgrade them so often?  In fact, I think that Magic Lantern fulfills a mission in that respect - by giving new life to older cameras it helps protecting our environment.  Another reason why I like it so much and why I am so gratefull to all our developers - the real missionaries!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 05, 2017, 05:11:24 PM
@IDA_ML - I'm not arguing with you, just hinting that it would be easier if we all had 5D3's. My first ML capable camera was an EOSM. I got it because thought it could do almost everything the other ML capable cameras could do. I does, depending on your definition of almost.

Back on topic--

Something interesting -- if you take Edmac#16 (on 7D) buffer "4410x1225"
4410/14*8 = 2520
So this could be your custom buffer 2520 x 1127 -- this is how I came up with my buffer.

How did you get 1127 from 1225?

What I'm looking for is how to translate this from the 1100D to the 7D and other cameras that use LVState.

Code: [Select]
#ifdef CONFIG_1100D
#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
#endif

Note that originally it was coded like this:

Code: [Select]
#define RAW_LV_BUFFER_ALLOC_SIZE (3906*766)
"Blame" shows that G. Condello (a.k.a. @nanomad) wrote that line but I searched all over and couldn't find out how he got the "3906" part. The "968" part is spelled out in this old comment you already posted:

Code: [Select]
408  * How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP:
409  *   - Go to LV and use lv_save_raw
410  *   - Check the RAW_LV_EDMAC debug info
411  *   - Suppose it reports W: 0xA3A H: 0x3C7 (taken from 1100D in LV mode)
412  *   - EDMAC W is the number of bytes per line
413  *   - The W resolutions is computed as: W * 8 / 14 (raw buffer is 14 bits per pixel)
414  *   - Thus 0xA3A  8 / 14 -> 1496 pixels
415  *   - EDMAC H is the number of "jumps"
416  *   - The H resolutions is H + 1 -> 0x3C8 -> 968 pixels

So is W width and H height of one of the buffers? Maybe but "W: 0xA3A H: 0x3C7" would translate to 2618x967. Note that the 1100D specifications show a maximum video resolution of "720p HD video at 25 or 30 fps" so maybe, just maybe, that's a reason why it didn't even need CONFIG_EDMAC_RAW_SLURP for it to work in the raw_video_10bit_12bit branch.

I also went back into the history of the source code and couldn't figure out when these "W:" and "H:" values were reported in hex but maybe (and this is just an uneducated guess) the values for the 7D should be taken from the maximum size of channel 16:

Code: [Select]
#ifdef CONFIG_7D
#define RAW_LV_BUFFER_ALLOC_SIZE (4410*1226)
#endif

Note the +1 on the "H:" value as noted in that old comment we are referencing.

This seems logical because the 7D can record higher resolution than the 1100D. Then again so many things in ML aren't what they seem so I might be totally off base on these assumptions.

By the way, your calculation of the width of the "custom buffer" matches up with the zoom mode raw buffer. I shot silent stills in various modes and this is what exiftool is showing on the DNG files:

Code: [Select]
mv640
Default Crop Size               : 1736 694
Active Area                     : 26 256 720 1992

mv720
Default Crop Size               : 1736 694
Active Area                     : 26 256 720 1992

mv1080
Default Crop Size               : 1736 1156
Active Area                     : 26 256 1182 1992

zoom
Default Crop Size               : 2520 1200
Active Area                     : 26 0 1226 2520

FRSP
Default Crop Size               : 5202 3466
Active Area                     : 50 158 3516 5360

Note the height of the zoom full raw buffer? Back to my question, "How did you get 1127 from 1225?" Would you agree that it should be 1226? That would mean that your "custom buffer" is really just the zoom mode buffer.

Exercise for other Digic IV cameras: Shoot silent DNG's at various video modes. Just change the Canon movie settings and shoot the silent stills with the camera in movie mode. Don't forget to do one for zoom (a.k.a. 5x crop mode). If anyone wants to try this but hasn't mastered exiftool, post a link to your DNG's and we'll check them out for you.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on December 05, 2017, 07:21:20 PM
RAW_LV_BUFFER_ALLOC_SIZE answered here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg188572#msg188572). You need to allocate space for how much raw image data *you* plan to save from connection #0 (you are reconfiguring the LiveView resolution, after all). For experiments, you could always allocate one SRM buffer, which is the largest contiguous chunk you can get from Canon's memory allocators - a 14-bit full-res uncompressed*) image won't be larger than that. This is done on 5D3 for full-res LV (crop_rec_4k).

*) "14-bit uncompressed" is important; it can get larger if you request 16-bit data, or after lossless compression (yes, a compressed image *can* be larger than the input image). Currently there's no known way to store such image in a contiguous chunk of memory. For lossless, it usually works, because - in practice - most of the time, the compressed image ends up smaller. Still, you also have to handle the case where compressed is larger than uncompressed (otherwise you'll overwrite other stuff in memory, sooner than you may think).

1100D *has* CONFIG_EDMAC_RAW_SLURP (check the build log from the Experiments page). Don't remember anyone reporting back for this model, so I don't know whether the 10/12-bit build works or not (or whether to commit the allocate-raw-lv-buffer merge, for that matter).
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 05, 2017, 08:52:36 PM
Thanks for clearing things up though at the same time a few things got muddied up.

1100D *has* CONFIG_EDMAC_RAW_SLURP (check the build log from the Experiments page). Don't remember anyone reporting back for this model, so I don't know whether the 10/12-bit build works or not (or whether to commit the allocate-raw-lv-buffer merge, for that matter).

So this whole thing about using the 1100D as an example might just turn out to be fool's gold? I could have sworn that raw_video_10bit_12bit 1100D/internals.h (https://bitbucket.org/hudson/magic-lantern/src/870cedc16e2d5ad35852cc8c7b04fd95ea020c45/platform/1100D.105/internals.h?at=raw_video_10bit_12bit&fileviewer=file-view-default) *doesn't* have CONFIG_EDMAC_RAW_SLURP defined while allocate-raw-lv-buffer 1100D/internals.h (https://bitbucket.org/hudson/magic-lantern/src/5d168ad4e8c9f6876ad33383eadf628218854c91/platform/1100D.105/internals.h?at=allocate-raw-lv-buffer&fileviewer=file-view-default) *does*. Looked at it again and I still don't see it. I also looked all over the build log (https://builds.magiclantern.fm/jenkins/job/raw_video_10bit_12bit/26/consoleText) and that didn't didn't turn up any nuggets.

However, this turned on a light:

RAW_LV_BUFFER_ALLOC_SIZE is how much we need for a full-res image (max W * max H * 14/8).

Looking at the exiftool output for the active area of a full-res silent image, 5360 * 3516 * 14/8 = 32980080 or 0x1F73C70 hex. Hum, sounds familiar:

Code: [Select]
#define SRM_BUFFER_SIZE 0x1F80000   /* print it from srm_malloc_cbr */
So does that mean something like this should work?

raw.c
Code: [Select]
#ifdef LV_STATE
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE)
#endif

Haven't tried it and my syntax might be out of whack but that's the idea.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on December 05, 2017, 09:11:02 PM
Another tip: each autoexec.bin has human-readable metadata - open it with nano and check the header. That tells you exactly where it was built from (and whether there were any uncommitted changes).

SRM_BUFFER_SIZE would work if you allocate with srm_malloc_suite. With the malloc wrappers, there will be a small overhead from the memory backend, so... just reduce the size by 0x100 bytes or so. Didn't try, but this should be generic for any model that's going to implement full-res LiveView.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 05, 2017, 10:01:46 PM
Quote
How did you get 1127 from 1225?
Sorry , mistake I meant 2520x1225 . But it seems I may be wrong after reading a1ex posts need to investigate more
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 05, 2017, 11:25:17 PM
Don't forget about this:

Code: [Select]
416  *   - The H resolutions is H + 1
That means it should be 2520x1226 and that works but we can go much further than that.

SRM_BUFFER_SIZE... just reduce the size by 0x100 bytes or so.

Did some experimenting and this is working:

Code: [Select]
#ifdef CONFIG_7D
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif

The generic "#ifdef LV_STATE" didn't work and that's the least amount of shaving off of SRM_BUFFER_SIZE that worked.

Let me qualify how well this allocat-lv-raw-buffer experiment is currently going. Up until now I was just checking DNG frames extracted from RAW and MLV files and they were looking good so I thought I'd render a video file and it was doing the old shake rattle and roll. In other words the DNG images looked clean but it appeared to be changing the capture area randomly. I backed out all the changes and guess what, it was still shaking so whatever I'm doing didn't cause this and didn't make it any worse.

So is CONFIG_EDMAC_RAW_SLURP working? Maybe, I don't know. Looks like the thing to do is to switch over to the raw_video_10bit_12bit branch and see what happens.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2017, 12:25:27 AM
Cool , I'll play with this on 5d2 and see what happens
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2017, 02:24:49 AM
Try out the code on the 5D2 , no problem every thing work normal in 1:1 & crop mode -- recorded 2 short .raw file 
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif
At least no ill effects , complied it on the iso_research branch I'm using , but did not load adtg_gui yet to see if it changes anything.

Edit: going though the DM-Spy Log from the 5D2
Code: [Select]
E05D3> GuiMainTas:ffa4522c:18:03: (PUB) SetImageWorkMemory 1:0x40d00000, 2:0x41700000
E060A> GuiMainTas:ffa45274:18:03: (PUB) SetImageWorkMaxPixel W:2200,H:1872,0x5e4340
E0640> GuiMainTas:ffa44db8:18:03: (PUB) SetEffectiveSizeOfJpeg(w:7488 h:4992) 2995
E066D> GuiMainTas:ffa44dd0:18:01: SetEffectiveSizeOfJpeg w:7488 h:4992

So can I assume that 0x5e4340 is 2200x1872 ?

I'm giving "RAW_SLURP" another try -- that why I was thing 0x5e4340 may work for that like this ?
Code: [Select]
#ifdef CONFIG_5D2
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5e4340))
#endif

I add
Code: [Select]
#define CONFIG_EDMAC_RAW_SLURPin to platform/5D2.212/internals.h
did I miss anything ?

I'll update later tonight.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 06, 2017, 04:37:33 AM
Houston, we have a problem. This is using a nightly build:

[EDIT] Nothing to see here -- discovered it was a post production problem. Camera is fine.

--snip--

Pressing on, I uploaded allocate-raw-lv-buffer branch builds to my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Included are builds for the 7D, 5D2, 50D, 550D and even the 1100D. Yeah, I know, not very interesting. It just seems to be an old branch that doesn't even have mlv_lite but it works quite differently on these cameras.

@reddeercity - I have no idea what you are doing but interested in how your experiment turns out. This is what I did with the allocate-raw-lv-buffer branch:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff

[EDIT] Got it working! 10bit/12bit on the 7D. I've got to do some more cleanup on the code because it is messing up the Digic V cameras but thought I'd put out the good news. Since there are so many requests from 7D users I uploaded an early build (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) if anyone wants to be a volunteer guinea pig.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on December 06, 2017, 10:14:00 AM
You got it working!!??? Maybe getting it for the 5D2 is similar. How did you do it??

I see your commit has CONFIG_EDMAC_RAW_SLURP defined for 5D2, could it be working yet?

Surely it can't be that far if you found it for 7d already, the difficult camera :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 06, 2017, 10:14:15 AM
dfort with the magic fingers. 7D working with 10/12 bits, tested just now:
10bit sample
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M06-1005_samples_10bit_7D.zip
12bit sample
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M06-1006_samples_12bit_7D.zip
14bit sample
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M06-1007_samples_14bit_7D.zip



Any chance to get a build for testing on 550D?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Andy600 on December 06, 2017, 10:48:12 AM
@dfort and for the 50D if you can.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on December 06, 2017, 12:42:31 PM
Dfort, you fully deserve your "Hero Member" status!  I can't wait to test.  Thank you so much!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2017, 02:59:34 PM
Ok can we please stay on topic here , any 10-12bit 7d should be discuss
either on the 7d or 10-12bit thread no here . This is for UHD/3.5k development.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 06, 2017, 03:44:35 PM
Code: [Select]
Ok can we please stay on topic here , any 10-12bit 7d should be discuss
either on the 7d or 10-12bit thread no here . This is for UHD/3.5k development.
I would rather welcome the guy(dfort) who potentially could hold the key to the raw_slurp code snippet for the 5D mark II. But hey, that´s just me.
Keep up the good work everyone. 10bit on 7D is golden.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: nikfreak on December 06, 2017, 04:33:37 PM
title says it all: "..and other digic IV cams"  and I bet the 7D will fit 3K continuous recording too :P
As far as I understand the LCD doesn't freeze now any more while recording 10/12bit @danne??
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 06, 2017, 04:59:28 PM
title says it all: "..and other digic IV cams"  and I bet the 7D will fit 3K continuous recording too :P
As far as I understand the LCD doesn't freeze now any more while recording 10/12bit @danne??

No freezing, no corruption, only happy, healthy MLV files  :D

*update. mv1080p mode that is
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on December 06, 2017, 05:43:48 PM
Try 720p60 in Canon menu with crop mode ... Frozen LCD here ...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2017, 06:39:28 PM
I'll ask nicely one more time , please stay ON TOPIC !!!
If that can not happen then I will remove this thread all
Together , as I started this thread and all information will be
Lost .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 06, 2017, 07:05:28 PM
This is for UHD/3.5k development.

Right, like I said before:

There are lots of parts to this.

You can continue trying to increase the image size but you won't be able to pass all that data to the card unless you reduce the bit depth and compress it.

This seemed like the best place to discuss the features needed to get Digic IV cameras recording at the highest possible resolution but since you feel I'm off topic I'll move the 10bit/12bit discussion somewhere else.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2017, 08:21:50 PM
Thanks, I agree there many parts & compression will be needed
At some point but not right now . I what to focus on max resolution
& custom buffers . I just don't what HD resolution 10-12bit discuss here
It better on the 10-12bit thread
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 06, 2017, 09:09:21 PM
Just as a suggestion, you probably should get the 5D2 working with the "basic" CROP_PRESET_3x3_1X before attempting the more advanced stuff. I helped with that on the EOSM, 700D, 100D and work is in progress on the 650D and 6D.

Whatever path you take to get to higher resolution on these older Digic IV cameras, best of luck!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Audionut on December 07, 2017, 01:32:37 PM
Hats off to you reddeercity.  You've gone from a 'typical' user to someone willing to dip their toes in the water.  I was excited to see you start this thread, but so busy at the time that I never got around to jumping in and sending some praises your way.  The funny thing about coding for ML is that no one really knows what they are doing.  Some people can for sure code better then others, and some people have more knowledge regarding some things then others, but it's all trial and error to begin with.

My advice would be to let the conversation flow, even when it begins to go a little off-topic.  Even in the off topic discussion, you're likely to find pieces of the specific part of the puzzle you are trying to solve, and sometimes it's just best to move onto another area of the puzzle, and then come back to the original area of the puzzle after you've had some success elsewhere.  I also prefer to split discussions that become large and deserve their own thread, rather then one or two posts.  So if the off-topic drags on for to long, send a pm and I'll happily split it.  And try to remember that we're all working towards the same end goal.  dfort isn't here to ruin the party, take all of the accolades or all of the attention, he seems to have some good amount of spare time atm, and he also seems to be dumping all of that time into ML.  This is a good thing for the project, and in my eyes, the project always comes first.  Take advantage of his time and knowledge, bug him about coding stuff.  Bug him for knowledge.

And the guys with all the know how need to remember that things that might seem easy, or common sense, generally aren't so easy for us plebes. (https://www.urbandictionary.com/define.php?term=plebe)  And sometimes we're happy just to plod along at our own slow pace, despite the common sense.  If that's frustrating for you, well, what goes around comes around.   :P

I get email notification about PM and forum reports, and always check emails.  I can find time if I'm really needed.  I'm here to help.  If you need to vent, pm me, make it obvious you need to vent.  Then vent.

And lastly, for the sake of clarity, deleted threads only disappear from a users point of view, mod+ have access.  Keep that in mind if you accidentally delete something, it's not gone, just pm me with the details.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on January 07, 2018, 02:16:51 AM
Was anything happening with lossless compressed raw on 550D?
And I can't find the main threads on compressed raw anymore, was it ever applicable to anything less than 14bit?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on January 07, 2018, 02:35:49 AM
And I can't find the main threads on compressed raw anymore

Top of page -> Downloads -> Downloads (pulldown menu) -> Experiments -> 4K raw video recording; lossless compression -> Forum discussion
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 26, 2018, 06:06:26 AM
@a1ex when you found the 3072 vertical resolution for the preset in this code from adtg_gui.c-1121 (https://bitbucket.org/hudson/magic-lantern/src/50d8f06018dc52f72f04265432dd83106e495e85/modules/adtg_gui/adtg_gui.c?fileviewer=file-view-default#adtg_gui.c-1121)
Code: [Select]
static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6804:                /* C0F06804 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
        }
    }
In my code I chanced "0x60804"(digic v) to "0x6084"(digic iv) before I test it l was hopping to find the right size for the 3k preset as you have "0x52801AB"
was this found thought testing different registers or for investigating Log files ?
I wondering if this is the reason I'm lock to resolution 3584x1068 .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on January 26, 2018, 08:51:51 AM
I've used trial and error at first, then figured out the math.

From raw.c, raw_lv_get_resolution: you have 0xC0F06800/4 <-> 0xC0F06084/8.

Vertical units (high half of the 32-bit value) are pixels; horizontal units are probably number of column groups (some columns are probably read out in parallel, so you can't get an arbitrary number of columns; it has to be multiple of column_factor = 2 on 5D2 and 8 on 5D3).

For the full-res LiveView, I tried to mimic the photo mode configuration. That didn't work out of the box, so I've tried to extend the resolution in small steps, until hitting some barrier (such as getting invalid image data or black bars past some resolution); then I've tweaked other registers that seemed to be size-related, until breaking these barriers. The option to diff video modes from adtg_gui was very helpful; so was the option to lock the modified registers, effectively attempting to turn one video mode into another; then, from these modified registers, I've tried to find a minimal subset that does the job (figuring out the meaning of these registers in the process). That's the general workflow; here's a step by step example (http://www.magiclantern.fm/forum/index.php?topic=10111.msg191218#msg191218).

Math for the above value:
- default raw buffer size on 5D3: 2080x1318 (see this post (http://www.magiclantern.fm/forum/index.php?topic=16054.msg195213;topicseen#msg195213), in particular, this file (http://a1ex.magiclantern.fm/bleeding-edge/raw/raw_res.txt))
- in 1080p (the video mode I've started from), the values of these two registers are 0x10017 and 0x528011b => 2080x1319;
  had to subtract 1 to address an issue with invalid frames; explained here (https://bitbucket.org/hudson/magic-lantern/pull-requests/897).
- out of these 2080 pixels, 146 are skipped from the left (optical black) and 2 from the right; that leaves 1932 active pixels;
  rounded to multiples of 8 to work on whole raw_pixblock's => 152+8 skipped, 1920 left
- (0x1AB - 0x17) * 8 - 160 = 3072.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 26, 2018, 10:10:45 AM
Didn't read the previous post correctly
sorry typo , It should had been 0x6804 for the source .
Thank you , That helps clear up things  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 27, 2018, 06:23:13 AM
I think I may have proof now that my custom buffer is correct for Full Frame (5792x3804 -- includes out of bound area) in Video Mode see screen shot below

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fpreview.ibb.co%2FmtzyOb%2FVRAM11_full_live_view_buffer.png&hash=52f0ebfc30284587639d62c89da368a0) (https://ibb.co/hQeyOb)

I did some logging with one of my memspy build a few months back with some screen shots , never really noticed it before until now.
in the red box you see resolution changed -- in the yellow box is the buffer size 2040x1267 -> in the green box 5792x3804 is the sensor size including out of bound area .
this buffer size (2040x1267) is what I use to get 3584x1068 in video mode with the my really , really ugly hack of the 5d3 3k preset .
I guess I can assume the size is correct and this here (http://a1ex.magiclantern.fm/bleeding-edge/raw/raw_res.txt) seem to back it up , 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on January 27, 2018, 07:57:30 AM
@reddeercity,
I've set up my Compiling System via dfort Mac tutorial, also clone dfort and I've test to see if my building Build work and every thing look like its work. Only thing I change was in raw.c was
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif

to

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#endif

by the way is that correct

I'm going to test it now, but what can I do to help out or get to your latest stable build?

by the way when I put code below in Terminal it gave me 2 build

cd Magic-LanternCloneRepository/platform/5D2.212
make zip
cp *.zip ~/Desktop
make clean
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 03, 2018, 05:51:58 AM
Came across some Ti (Texas instruments) chip info from a screen shot , it's the raw pipe with internal 10 & 12 bit
this is helping me understand what going on , for digic iv UHD

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FnHhUER%2Fcmos2raw_flow.jpg&hash=f67fd4a25002b47abf776f0210f7ff25) (https://ibb.co/nHhUER)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 10, 2018, 03:33:49 AM
Reading my decompile rom in Microsoft  Visual Studio , very nice to use  :)
For some info found this
Code: [Select]
ff0322b4 > LimitRawSize [%d]
So I guess there must be some limits for processing image , would be nice to know what it is
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 10, 2018, 03:41:34 AM
Some liveview raw stuff from rom I think
Code: [Select]
ff034b80 Enable AE Sequence = %1d
ff034b9c Enable AF Sequence = %1d
ff034bb8 Output AF RAW Data = %1d
ff034bd4 Output AE RAW Data = %1d
ff034bf0 Output DEFM Raw    = %1d
ff034c0c Output RSHD Raw    = %1d
so when the af & ae sequence are enabled there are output to more then one regs
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on March 22, 2018, 12:02:29 PM
I try to increase the size of 5D2 and i have some success without using adtg_gui.mo.
As soon as possible I hope come back  to post all my work on the 5D2.
For the instant I have a problem for increase height.

For Digic register like  C0f06084/88  i use EngDrvOutLV(digic_register, new_val) and it' s OK . The new value is correctly apply.

But i need to tweak C0F0713C/50 too for increase height , so i tried different solutions but the value of digic register don't change :

* in ROM1.BIN.dis (5D2)
   With C0F0713C for example:

Code: [Select]
ff186ee4: e51f0064 ldr r0, [pc, #-100] ; ff186e88: (00007d9c)
ff186ee8: e59f11c8 ldr r1, [pc, #456] ; ff1870b8: (c0f0713c)
..................
ff986ee4: e51f0064 ldr r0, [pc, #-100] ; ff986e88: (00007d9c)
ff986ee8: e59f11c8 ldr r1, [pc, #456] ; ff9870b8: (c0f0713c)

So i believe understand that  instructOri = e59f11c8 and try to increase it with InstructNew= e59f11dc
With this change I hope to transform  ldr    r1, [pc, #456]         in        ldr      r1, [pc, #476]   but this don't  works.

It is possible with adtg_gui.mo but there is too much troubles , please someone could explain me how to write in this registers (c0f0713c/50) ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 05, 2018, 10:28:35 PM
Hey waza57 , cool  :)
can you share some more details ? looks interesting -- can you save a dng?
I can't answer your question about "write in this registers"
I did all my stuff with the iso_research branch , which included 3k preset with crop_rec (1:1)
I didn't have any problems or issue with adtg_gui.mo , what was the problem , enabling changed registers valves ?

Found a tool to help with find fps head timer numbers , thou it have more uses then that
eostimergen (http://www.g3gg0.de/wordpress/projects/eostimergen/)  :)) Thanks @g3gg0

Quote
EOSTimerGen is a tool for Magic Lantern users that want to
 a) over/undercrank the camera (to get higher or lower frame rates)
 b) fine tune their frame rate to overcome production tolerances
 c) synchronize two or more cameras to have nearly the same rate


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FeZeAxH%2FEOS_Video_Time_Gen.png&hash=10404549f53ef4172f9d63158e69d8ca) (https://imgbb.com/)

You can do that with frame over ride but nice to do it in software to check it .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on April 16, 2018, 09:44:44 PM
reddeercity, Thanks for that but it seems timer B don't increase height like with other models.

I tried a lot of things but nothing works.

Perhaps ADTG [8xxx] but unfortunately I don't succeed to see it with adtg-gui module in 5D2.

An example :
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2Fb79O2S%2FM17_0638_00000.png&hash=9d1f291f5b3e1d309ce33815cae2d807) (https://ibb.co/b79O2S)

how to delete mint account (https://deleteacc.com/mint)


effective width: 2840
effective height: 1082

http://www.adapteo.fr/images/M17-0638_000000.dng (http://www.adapteo.fr/images/M17-0638_000000.dng)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 16, 2018, 10:40:16 PM
OMG 😍😍😍!!! waza57 your the person that made video dual ISO on 5D2 almols work rigtht??!

Can you record MLV yet with this confuguration? If not, how close is it??? And how could I help? The goal would be to put it in a single menu option in a normal build, how much work is that? I would love to help if possible. Or test it  ;) ;) ;) ;) ;) (as every noob always says)

I'm so excited!!! 2.8K—that's Alexa sensor resolution (it must be a sign)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 17, 2018, 12:47:43 AM
reddeercity, Thanks for that but it seems timer B don't increase height like with other models.
I tried a lot of things but nothing works.
Perhaps ADTG [8xxx] but unfortunately I don't succeed to see it with adtg-gui module in 5D2.
try maybe ADTG1 [105f]N , or the ADTG2 -- one is for 3xcrop mode & the other is for 1:1 FHD
I've had some success increasing a little height
I most I've been able to get is 1068(h) with the (w) 3584
I see the frame rate is 23.614 with exiftool , this maybe the issue -- the other test I've seen on digic4 have frame rate at 12fps
Have you tried to slow the frame rate down ? Did you try with frame override enabled or disabled ?

Very interesting @waza57 -- can you share your code yet ? I would like very much to look at it , if you don't want to publicly display it pm me  :)

I've being going with just modifying CMOS 1 & 2 registers with the Crop_Record Module loaded , it seem to put the sensor in 1:1 pixel mode centered
then enable the 5d3 3K preset ( the reg's are very close to 5d2 and it kind of works , I have a timer problem now) 
Did you get this in 3xCrop mode or 1:1FHD ?
Good job  :))

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 20, 2018, 06:36:49 AM
@ waza57 great ! you got your code up crop_rec_4k_5D2  (https://bitbucket.org/waza57/magic-lantern/commits/f73a13b72e7fb4e7757ab6547a7685d6dcbc5216) -- I see you are extending the 3xCrop windows , interesting then for sure look at the those  ADTG1 [105f]N , &  ADTG2 in the adtg_gui.mo
I played around with then in 3xcrop_mode .
If this so , then the old  10bit without raw_slurp will work in 3xCrop_mode with that extended resolution .
I coming from it thought 1:1 FHD with the crop_record.mo -- it put the sensor into every line , by passes the line skipping ,
I'll have a closer look at the code  , maybe we can have a special "4k crop_record.mo " for digic4  ;) 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on April 23, 2018, 11:22:30 PM
@reddeercity have you tested the 2.8k yet??? I really wanna see some nice samples of 5D mark ii 2.8k!!! It will give me motivation to fix my broken 5D2

I'm truly excited to see if anything finally happens with this camera.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 24, 2018, 06:53:59 AM
@Ilia3101 no not yet , but I'll give it try soon.
I'm actually trying to find out why HDMI is broken in the nightly builds (unified) no one has reported it :(
and as far as I can tell it's been broken since the "New Core" (I use the Old Core for production work I must have
HDMI for my production workflow) , so I'm undecided to which way to go . I can implement 10-12bit
and my experiments in UHD plus waza57 code in to the "Old Core" I'm starting to see too many restriction with the new core as
it must be compatible with Digic 5 Camera (5D3) and I'm not to sure about the Digic's 4 (5D2) life expectedly or support .
I know it sound discouraging , but I have to wonder if all this time & effort is really worth it after all .
I can deal with a limited 1:1 resolution but not HDMI , that the deal breaker .
So I need to recess this thread in the direction it goes .
Sorry to be off topic a bit .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 25, 2018, 07:02:22 AM
reddeercity, Thanks for that but it seems timer B don't increase height like with other models.
I tried a lot of things but nothing works.
Perhaps ADTG [8xxx] but unfortunately I don't succeed to see it with adtg-gui module in 5D2.
An example :
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2Fb79O2S%2FM17_0638_00000.png&hash=9d1f291f5b3e1d309ce33815cae2d807) (https://ibb.co/b79O2S)
effective width: 2840
effective height: 1082
Being trying to reproduce this from @waza57 but no go , I clone crop_rec_4k_5d2 branch and update to comment f73a13b (https://bitbucket.org/waza57/magic-lantern/commits/09dfad1ad108fbe480077aab0bf39f418cc2cda2) -- I can load crop_crop.mo but it's disable , the mlv_lite and adtg_gui.mo didn't make .
So can you tell me how you got this please ? was this on mlv_lite or mlv_rec ? I see in the "Digic Poke" where you code is , can you share the setting in there ?
In the mean time I'll play with more , maybe by chance i can find the right setting to reproduce this  :D
Edit: your working in this right ?
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FfufTmx%2FVRAM0.png&hash=c634b936e3089c31a5c2c0fca175cda9) (https://ibb.co/fufTmx)
Edit:2 it seem I'm still missing some code in raw.c 9a0b318 (https://bitbucket.org/waza57/magic-lantern/commits/9a0b318953a02cc12adc09e44253c5772abeb939) plus some comment's I missed , I thought the f73a13b (https://bitbucket.org/waza57/magic-lantern/commits/f73a13b72e7fb4e7757ab6547a7685d6dcbc5216) comment was the consolidation of the code from waza57 , I'll continue tomorrow


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 26, 2018, 08:03:18 AM
I'm getting some error compiling now in raw.c , I'm not sure why
Code: [Select]
../../src/raw.c: In function 'raw_lv_get_resolution':
../../src/raw.c:682:5: error: invalid storage class for function 'raw_update_params_work'
 int raw_update_params_work()

Code: [Select]
int raw_update_params_work()anything to do with "invalid storage class for function 'raw_update" etc. .....
Can some one please help explain this error .
I'm working off waza57 "crop_rec_5d2"
https://bitbucket.org/waza57/magic-lantern/commits/9a0b318953a02cc12adc09e44253c5772abeb939#Lsrc/raw.cF678T678
Edit: Here my source code I'm working off I uploaded to my bitbucket downloads
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2018-4-26-crop_rec_5d2.zip
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 03, 2018, 07:53:47 AM
I put aside "waza57 crop_rec_5d2" code on hold and when back my code and my some improvements (It's gets better)  :)
in raw.c I added
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#end
In adtg_gui.c I changed 0x6800 -->0x6084/8
Code: [Select]
static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6084/8:                /* C0F06084/8 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
        }
    }
and added in mlv_lite.c the extended resolutions
Code: [Select]
3840, 4096

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FitnjpS%2FVRAM1.png&hash=07f882cf887a38594f428d9b8d9e082c) (https://ibb.co/itnjpS)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2Fb44aw7%2FVRAM5.png&hash=2f6a621b5e7ee9edb8257a944f6b0e8c) (https://ibb.co/b44aw7)
 
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2Fd4fyG7%2FVRAM4.png&hash=b061452bec4eb697b85c576928801999) (https://ibb.co/d4fyG7)

I was able to extend the horizontal to1076 from 1044 , not much but a start  :)
I did dump the image buffer this what I'm getting , (please remember I did not do any timer adjustment yet)
The HD Dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FnkzEpS%2FHD_000_Small.png&hash=2c8a6a5ed4d8d5177c5064372637fa16) (https://imgbb.com/)
The LV Dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fb51c9S%2FLV_001_Small.png&hash=14d2f03eff08ba67ef3b4be4d5ef7ba4) (https://imgbb.com/)
The second HD Dump after trying to move the image down to center
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjSa4pS%2FHD_001_Small.png&hash=8e47933455d6a7c9f8555fe81eedb3bd) (https://imgbb.com/)
Playing around with liveview preview
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fdw5yG7%2FLV_000_small.png&hash=80452f7b4c9422ddeddf601c045c32b1) (https://imgbb.com/)
Screen shot -- I didn't do any timer adjustment yet
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdeiJG7%2FVRAM3_Small.png&hash=d01812e396793aae86652b5f8244028a) (https://imgbb.com/)

Here the build I compiled (the latest updated source + ISO Research)
Code: [Select]
# Magic Lantern Nightly.2018May03.5D2212 (f683753bc4e3+ (iso-research))
# Built on 2018-05-03 03:54:46 UTC by ml@ml-pc
# Configuration saved on 2018/05/02 22:40:24
beta.warn = 2
menu.first = -4
movie.log = 1
rec.notify = 0
enable-liveview = 2
fps.preset = 1
fps.override.idx = 32
fps.override = 1
battery.drain.rate.rev = 59
hist.log = 0
spotmeter.draw = 0
clear.preview = 4
zebra.draw = 0
disp.mode.x = 149

So What I did was load the adtg_gui.mo ,  mlv_lite.mo  &  Crop_Rec.mo with crop_rec I put the cam in to 1:1 3x mode so every pixel & line (by pass line skipping)
then loaded the 3k 5d3 preset but I changed the regs as noted above .
It changed CMOS[1]0xc00 --> 0x2cb & CMOS[2] 0x40e --> 0x10e
I'm able to switch between all res's up to 4096 (I added 3840 & 4096) yes I can set it to 3520 also , before I was lock to just 3584 and thou the cmos[2]
I was able to change the  horizontal by 32 lines , (not much but a start)

I really just simplified the code after learning what I read in the 10-12bit thread  ;D
I think this is the right path , though the registers just like the 5d3 only make sense being there's so closely related -- brother from a different mother  :D
more to come , next is to clean the raw image up and start recording a image only if for a few second until the buffer is full then I need to finish the lossless compressed raw
code in silent .c
The last thing is the content of MENUS.CFG
Code: [Select]
0 3F77F0FE 3D41FF14 Overlay\Zebras
0 3F783078 3D556563 Overlay\Spotmeter
0 409A2D5E 3E935DBF Movie\FPS override
0 3FF3E2C6 3CA1335E Movie\Crop mode
0 409BA922 3F3448D6 Movie\RAW video
0 3F711935 3B573CCE Movie\Movie Tweaks
0 3FFA4E97 3E70270A Display\Clear overlays
0 407B0F0F 3F208C48 Display\Anamorphic
0 3FFB325A 3EF96A01 Debug\Memory patches
0 3FF8BDA4 3EF33695 Debug\Show console
0 409C1067 3F4B1C91 Debug\ADTG Registers
0 3F7308B9 3BE6B0C1 Debug\Lens info
0 40BC1075 40007E74 Debug\Screenshot - 10s
0 3FFA92E5 3EDB7170 Debug\Dump image buffers
0 3F7346F1 3BFDC26F Debug\Don't click me!
0 3F7E7825 3F10815C Debug\Free Memory
0 3F7156EE 3B6CC2E3 Movie Tweaks\Movie Logging
0 3F7194B7 3B8237FD Movie Tweaks\REC/STBY notify
0 3F71D290 3B8F3D97 Movie Tweaks\Force LiveView
0 3F72CA91 3BD1B7F5 FPS override\Desired FPS
0 3F728C79 3BBEA73B FPS override\Optimize for
0 3F787002 3D6ABC54 Histogram\Scaling
0 3F75B891 3CA48BF9 Advanced\DIGIC Registers
0 3F78EF46 3D8E03DC Advanced\Log Registers Now
0 3FF9750E 3EAE7395 Advanced\1:1 3K crop mode (5D3)
0 4079DAD8 3FA5BBBF RAW video\Resolution
0 3FF5D867 3D364505 RAW video\Aspect ratio
0 3FFF9DD2 3FDE2613 RAW video\Preview
0 3F747EF9 3C4C5756 RAW video\Advanced...
0 3F7EB94A 3F1EF4B2 Free Memory\Allocated RAM
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 04, 2018, 08:12:50 AM
some progress cleaning up Lv @ 4096x1044

Image dump of Liveview in "framing preview"
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fmxi5yn%2FLV_003_small.png&hash=9d929c5537349e86d47a989c4d67d855) (https://imgbb.com/)
adjustments CMOS[1]  with ADTG, 0x106E, Shutter blanking for LiveView 1x
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fkv0qW7%2FLV_002_small.png&hash=18f14a0ac235e8df49253dbe3b3fd748) (https://imgbb.com/)
Screen capture Liveview
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fhe6fW7%2FVRAM9_small.png&hash=77294568a071c22820661daa68be4634) (https://imgbb.com/)

Even thou it in 4096x1044 It seem it retaining the 1856 buffer & just mirroring  it
maybe my size is not right but with SRM buffer my understanding there should enough 32mb per frame , right ?
 
Code: [Select]
RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)I thing this what keeping me from saving a DNG on dumps , it does export a DNG but it's empty
I'll continue tomorrow .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 05, 2018, 06:00:11 AM
Still haven't got the raw image right yet but I can increase the vertical to 1250 , so that's 4096x1250 .
It's was a matter of just changing Aspect Ratio to 1:1 or 3:2 .
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhVPXeS%2FVRAM12_Small.png&hash=f1bba934582ecd1557eb92112accd6a8) (https://imgbb.com/)
Image dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fj21pR7%2FLV_005_Small.png&hash=4ad6ba2a0bed3d24050b94472043f8eb) (https://imgbb.com/)

Edit: Gave a go at 3x crop mode to see if I could get something there got a interesting result
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FciEeB7%2FVRAM16_Small_3xcrop.png&hash=a2082bb0d63ac683fa9c78461c3af122) (https://imgbb.com/)
Just played around with c0f0 0x6084 & c0f0 0x6088 reg's and when to the change the res in the raw video menu .
I don't think there any there yet but interesting

Edit:2 I thing I have to rework the buffer or I need to fully implement raw_slurp  with SRM memory
I can easily put the sensor in to 1:1 without the 3k preset in adtg_gui.mo
Code: [Select]
cmos[1] 0xc00 --> 0xbc0  (height)
cmos[2] 0x40e --> 0x10e (width)
With that I choose up to 4096x1250 , I got that by just switching in to 3x crop_mode and reading what the reg's at cmos 1&2  are.
plus I think I found the reg's for c0f0 0x6084 & 0x6088 in 3x crop_mode
Code: [Select]
c0f0 0x6084 --> 0x10036 (1:1 FHD)
c0f0 0x6084 --> 0x30036 (3x crop_mode)
c0f0 0x6088 --> 0x4f40432 (1:1 FHD)
c0f0 0x6088 --> 0x46a04ba (3x crop_mode)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 06, 2018, 06:26:27 AM
Having some trouble get the custom buffer to compile for the higher res redirect using srm memory  as per a1ex  talks said here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg183374#msg183374)  I found the comment  crop_rec_4k#Lsrc/raw.cT149 (https://bitbucket.org/hudson/magic-lantern/commits/7ad3754768b26a106d057550ca359089ba837fcf?at=crop_rec_4k#Lsrc/raw.cT149)  and I finished implmenting the rest of raw_slurp for srm memory as per what a1ex has done for the 5d3 for 4k . So my error is this
src/raw.c:578:39: error: 'DEFAULT_RAW_BUFFER'...........
see below -- from the terminal window
Code: [Select]
../../src/raw.c:161:0: warning: "RAW_LV_BUFFER_ALLOC_SIZE" redefined [enabled by default]
 #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
 ^
../../src/raw.c:147:0: note: this is the location of the previous definition
 #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
 ^
../../src/raw.c: In function 'raw_lv_realloc_buffer':
../../src/raw.c:578:39: error: 'DEFAULT_RAW_BUFFER' undeclared (first use in this function)
         if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
                                       ^
../../src/raw.c:578:39: note: each undeclared identifier is reported only once for each function it appears in
make: *** [raw.o] Error 1
if you have trouble following what posted from the raw.c I uploaded the whole file to my bitbucket downloads folder
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/adtg_gui_4k_raw_slurp_redierct_buffer-raw.c
here  part of the code, I hope someone can help me understand the problem
I added more code then needed  so everyone can get the idea of with happening

 
Code: [Select]
71- /*********************** Camera-specific constants ****************************/
 72-
 73- #ifdef CONFIG_EDMAC_RAW_SLURP
 74- /* undefine so we don't use it by mistake */
 75- #undef RAW_LV_EDMAC
 76-
 77- /* hardcode Canon's raw buffer directly */
 78- /* you can find it from lv_raw_dump, arg1 passed to dump_file:
 79-  *
  80- * raw_buffer = get_raw_buffer()
  81- * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
  82-  * ...
  83-  * dump_file(filename, raw_buffer, 7*something...)
  84- */
 ..................
146- #ifdef CONFIG_5D2
147- #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
148- #define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024)     (I added this as I thought it be big enough , not too sure thou -- is this my problem  not sure how to find it out )
149- #endif
150-
151- #ifndef DEFAULT_RAW_BUFFER_SIZE
152- /* todo: figure out how much Canon code allocates for their LV RAW buffer - how? */
153- #warning FIXME: using dummy DEFAULT_RAW_BUFFER_SIZE
154- #define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024)
155- #endif
156-
157- /* for higher resolutions we'll allocate a new buffer, as needed */
158- /* all cameras using CONFIG_EDMAC_RAW_SLURP should be able to handle this */
159- /* SRM_BUFFER_SIZE matches the full-res image size, as 14-bit uncompressed (actually a bit larger, but not much) */
160- #define CONFIG_ALLOCATE_RAW_LV_BUFFER
161- #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
..........................

 549- #ifdef CONFIG_EDMAC_RAW_SLURP
 550-
 551- /* requires raw_sem */
 552- static void raw_lv_free_buffer()
 553- {
 554-   printf("Freeing LV raw buffer %x.\n", raw_lv_buffer);
 555-   if(raw_allocated_lv_buffer) {
 556-        free(raw_allocated_lv_buffer);
 557-        raw_allocated_lv_buffer = 0;
 558-   }
 559-   raw_lv_buffer = 0;
 560-   raw_lv_buffer_size = 0;
 561- }
 562-
 563- /* requires raw_sem */
 564- static void raw_lv_realloc_buffer()
 565- {
 566-   int width, height;
 567-  int ok = raw_lv_get_resolution(&width, &height);
 568-  if (!ok)
 569-  {
 570-      ASSERT(0);
  571-     return;
 572-   }
 573-
 574-    int required_size = width * height * 14/8;
 575-    if (DEFAULT_RAW_BUFFER_SIZE >= required_size)
 576-   {
 577-      /* no need for a larger buffer */
 578-      if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)     ----- (this where the error is  not sure to resolve it )
 579-      {
 580-          printf("Default raw buffer OK for %dx%d (%s)", width, height, format_memory_size(required_size));
 581-
 582-          if (raw_lv_buffer && raw_lv_buffer == raw_allocated_lv_buffer)
 583-          {
 584-             printf(" - back to default.\n");
  585-            raw_lv_free_buffer();
  586-         }
  587-         else if (raw_lv_buffer)
  588-         {
  589-               printf(": %x -> %x\n", raw_lv_buffer, DEFAULT_RAW_BUFFER);
   590-        }
  591-         else
  592-         {
  593-              printf(".\n");
  594-           }
   595-       }
   596-
   597-      raw_lv_buffer = (void *) DEFAULT_RAW_BUFFER;
   598-      raw_lv_buffer_size = DEFAULT_RAW_BUFFER_SIZE;
   599-      return;
   600-   }
   601-
   602-    if (raw_lv_buffer_size >= required_size)
   603-    {
   604-          /* no need for a larger buffer */
    605-       return;
    606-    }
    607-
    608-   printf("Default raw buffer too small (%s", format_memory_size(raw_lv_buffer_size));
    609-   printf(", need %dx%d %s) - reallocating.\n", width, height, format_memory_size(required_size));
    610-
    611-   if (raw_lv_buffer && raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
    612-   {
    613-       ASSERT(0);
    614-        return;
     615-   }
    616-
    617- #ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
    618-         raw_allocated_lv_buffer = fio_malloc(RAW_LV_BUFFER_ALLOC_SIZE);
    619-         raw_lv_buffer = raw_allocated_lv_buffer;
    620-         raw_lv_buffer_size = RAW_LV_BUFFER_ALLOC_SIZE;
    621-         return;
622- #endif /* CONFIG_ALLOCATE_RAW_LV_BUFFER */
623-
624-          /* you should enable CONFIG_ALLOCATE_RAW_LV_BUFFER
625-           * or find some other way to reserve memory for the RAW LV buffer */
626-          ASSERT(0);
627- }
628-
629- #endif  /* CONFIG_EDMAC_RAW_SLURP */
630- #endif /* CONFIG_RAW_LIVEVIEW */
631-
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 09, 2018, 01:00:11 AM
It look like I have problems with the redirect buffer
crop_rec_4k#Lsrc/raw.c-614 (https://bitbucket.org/hudson/magic-lantern/commits/7ad3754768b26a106d057550ca359089ba837fcf?at=crop_rec_4k#Lsrc/raw.cF616T614)
Code: [Select]
if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)I added
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024).
#endif
Is there a way to find the "DEFAULT_RAW_BUFFER_SIZE" ?
 I just copy the default values from
crop_rec_4k#Lsrc/raw.cT146 (https://bitbucket.org/hudson/magic-lantern/commits/7ad3754768b26a106d057550ca359089ba837fcf?at=crop_rec_4k#Lsrc/raw.cF154T146)
Code: [Select]
#define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024)
#endif
but if I'm using "SRM_Buffer" do I really need the "Default_Raw_Buffer_Size"
this is confusing me , I did find the "working buffer size" in one of my logs but seem too small
Code: [Select]
1726C> GuiMainTas:ffa4522c:18:03: (PUB) SetImageWorkMemory 1:0x40d00000, 2:0x41700000
172A7> GuiMainTas:ffa45274:18:03: (PUB) SetImageWorkMaxPixel W:2200,H:1872,0x5e4340

as it for 2200x1872 pixels.

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 10, 2018, 06:31:43 AM
Will every think works as far as getting in to 1:1 pixel mode in FHD and ml can select any resolution from 1856 to 4096 up to 1:1 Aspect Ratio 
and it will be able (theoretical specking) to capture up full width 5634 . Very clever software , Magic Lantern !
I starting to see the genius of the code -- the whole infrastructure is very flexible , once I followed a1ex's comments , it's all very clear now :))  ,
the 5d2 works so very close to 5d3 but with I small difference really , e.g. , c0f0 0x6800 0x6804 (d5/5d3) c0f0 0x6088 0x6084 (d4/5d2)
Indeed the 5d3's has the same cmos [2] reg's (0x10E) ,  is the same as the 5d2 cmos[2] reg's (0x10E) for 1:1 pixel FHD and the vertical offset cmos[1]  is a little different
but close enough to get started on cleaning up the offset .
The height limitations is equal to the height of the image buffer , in my case it's still using either the full HD buffer (1880x1250) in 1:1 or 3x crop 2144x1076 and that's the new kettle of fish Has to be solved .
I reproduce every thing in 3xcrop_mode and can get up to 4096x1076 , but the height is lock to the 3x buffer , that why I can get 1250 vertical in 1:1 because the image buffer size ,
the default full raw buffer size need to be found so the redirect buffer code infrastructure can be used with SRM Memory like all the rest of the cam or almost all .
If that can't work then a custom redirect image buffer for d4 cam will need to be written , maybe similar to what I was working on  before raw_slurp
Code: [Select]
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
that's equal to the full image buffer full image buffer (https://www.magiclantern.fm/forum/index.php?topic=19336.msg196437#msg196437) on 5d2 .

I have a request from any d4 cam users (50d ,550d etc. ..) but I prefer 50d or 7d , I'm interested in if we can put others d4 cams in to 1:1 FHD 4k mode
you need ADTG_GUI.mo from the ISO-Research branch , the best way is to compile with the latest source +the iso-research branch .
some small mod's to adtg_gui.c  adtg_gui.c-1127 (https://bitbucket.org/hudson/magic-lantern/src/50d8f06018dc52f72f04265432dd83106e495e85/modules/adtg_gui/adtg_gui.c?fileviewer=file-view-default#adtg_gui.c-1127) change 0x6804 to 0x6084/8 & C0F06804 to C0F06084/8
and in mlv_lite.c add 3840 & 4096 to the list of resolutions that sure do it. If you can't compile of has a problem let me know here and I can provide a build.
What I'm looking for is the cmos[1] , cmos[2] & c0f0 0x6084 & c0f0 0x6088 you'll need to enable digic reg's and put the cam in to 3x crop_mode and record the setting at those reg's
c0f0 0x6084 should be something like 0x30036 in 3x crop mode & 0x10036 in 1:1 mode at least that what the 5d2 says & cmos[2] should be 0x10e in 3x crop 1:1 is 0x40e same thing
that the 5d2 info so if other cam report the same then we can port UHD or at least a true crop record of the sensor in 1:1 maybe 2.5k so who knows.
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 14, 2018, 06:13:55 AM
Will I got the redirect buffer to compile in raw.c -- stupid mistake in the "#ifdef CONFIG_5D2" forgot to close the #ifdef with "#endif" :(
any how as I said before I implemented "raw_slurp" for "srm memory" for the redirect buffer . link to the raw.c file below if any one wants to follow along closer
redirect_buffer_4k_res_iso-research_branch_raw.c (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/redirect%20_buffer_4k_res_iso-research_branch_raw.c)
added
Code: [Select]
#ifdef CONFIG_5D2
/* for higher resolutions we'll allocate a new buffer, as needed */
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#define DEFAULT_RAW_BUFFER (4096*4096)
#endif

not sure if it working yet , just start to experimenting  , I don't think the "default_raw_buffer" is correct I just pick the biggest one there was I saw
in the EDMAC 4096*4096 . I did try 3574*1268 but when I enable raw vide(mlv_lite) all there was "0" or 0x0 for size so I changed to 4096*4096 and raw vide work again at
least before I enabled the adtg_gui.mo . Needed to be reworked more , I wanted at least at this point to be able too compile and it does so fine tuning  buffer sizes
and or finding the correct size is next .

the redirect buffer
Code: [Select]
/* requires raw_sem */
static void raw_lv_realloc_buffer()
{
    int width, height;
    int ok = raw_lv_get_resolution(&width, &height);
    if (!ok)
    {
        ASSERT(0);
        return;
    }

    int required_size = width * height * 14/8;
    if (DEFAULT_RAW_BUFFER_SIZE >= required_size)
    {
        /* no need for a larger buffer */
        if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
        {
            printf("Default raw buffer OK for %dx%d (%s)", width, height, format_memory_size(required_size));

            if (raw_lv_buffer && raw_lv_buffer == raw_allocated_lv_buffer)
            {
                printf(" - back to default.\n");
                raw_lv_free_buffer();
            }
            else if (raw_lv_buffer)
            {
                printf(": %x -> %x\n", raw_lv_buffer, DEFAULT_RAW_BUFFER);
            }
            else
            {
                printf(".\n");
            }
        }

        raw_lv_buffer = (void *) DEFAULT_RAW_BUFFER;
        raw_lv_buffer_size = DEFAULT_RAW_BUFFER_SIZE;
        return;
    }

    if (raw_lv_buffer_size >= required_size)
    {
        /* no need for a larger buffer */
        return;
    }

    printf("Default raw buffer too small (%s", format_memory_size(raw_lv_buffer_size));
    printf(", need %dx%d %s) - reallocating.\n", width, height, format_memory_size(required_size));

    if (raw_lv_buffer && raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
    {
        ASSERT(0);
        return;
    }

#ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
    raw_allocated_lv_buffer = fio_malloc(RAW_LV_BUFFER_ALLOC_SIZE);
    raw_lv_buffer = raw_allocated_lv_buffer;
    raw_lv_buffer_size = RAW_LV_BUFFER_ALLOC_SIZE;
    return;
#endif /* CONFIG_ALLOCATE_RAW_LV_BUFFER */

    /* you should enable CONFIG_ALLOCATE_RAW_LV_BUFFER
     * or find some other way to reserve memory for the RAW LV buffer */
    ASSERT(0);
}

In the edmac-memcpy.c -- added for raw_slurp (I just took the code from the  here (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff)
Code: [Select]
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP
   ......
#elif defined(EVF_STATE)
uint32_t raw_write_chan = 0x12; /* 60D and newer, including all DIGIC V */
#elif defined(CONFIG_5D2)
uint32_t raw_write_chan = 0x02; /* testing 0x01 and 0x02 */
#endif

Code: [Select]
platform/5D2.212/consts.h
#define SRM_BUFFER_SIZE 0x25D0000   /* print it from srm_malloc_cbr */
Code: [Select]
platform/5D2.212/internals.h
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP
#define CONFIG_ALLOCATE_RAW_LV_BUFFER
adtg_gui.c I changed c0f0 0x6804 to 6084/8
Code: [Select]
static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6084/8:                /* C0F06084/8 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
        }
    }

    return 0;
}
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 17, 2018, 01:50:19 AM
another piece of the puzzle of the redirect buffer
I just added
Code: [Select]
/* for higher resolutions we'll allocate a new buffer, as needed */
/* all cameras using CONFIG_EDMAC_RAW_SLURP should be able to handle this */
/* SRM_BUFFER_SIZE matches the full-res image size, as 14-bit uncompressed (actually a bit larger, but not much) */
#define CONFIG_ALLOCATE_RAW_LV_BUFFER
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
I updated the raw.c and it's on my bitbucket downloads I renamed it to adtg_gui_redirect_4k_buffer_raw.c (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/adtg_gui_redirect_4k_buffer_raw.c) for those who are following my progress .

I also found out how to find the "default_raw_buffer" -- compile the allocate-raw-lv-buffer branch with these comments from a1ex e799e89 (https://bitbucket.org/hudson/magic-lantern/commits/e799e89f67877825c6a4dc2868f87bed1196432b)
Quote
Raw backend -- attempt to autodetect Canon's raw buffer size
Usage:
- compile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (config-defines.h)
- open the console (Debug menu) and enable something that uses LiveView RAW features (raw video, raw histogram etc)
- test by starting the camera in all video modes (photo, 1080p, 720p, crop, x5 etc)
- take a screenshot or write down the console messages

This something all Digic 4 cam user can do easily and would help with other features beside 4k/UHD
It you do please just post result on this thread .
looking for :
5D2 -- (I should be able to get this one , but it won't hurt to post other results)
50D
7D
550d
600d (? maybe )
did I miss any ? for digic4
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on May 18, 2018, 08:20:22 AM
some progress in 3x crop_mode , did a image dump LV-005.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-005.422) LV-004.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-004.422) , starting to get the image to move to the left and push out the ghost image .
you can see the black border shrinking , tried to export the 422 image but the "YUV 422 Convertor" didn't recognizes the resolution but it could be view in the app.
most notable things I'm adjusting is the "A" &"B" Timers & the 3x shutter blanking & cmos[4]
screen shots of app with 422 dumps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fi88JAd%2F3xcrop_mode_3200x1076.png&hash=2e4637c346221656419422859200fcdf) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FiNtqOy%2F3xcrop_mode_3200x1076_adjustments.png&hash=0aaf6b1204e1e37f333bf3e171c81e13) (https://imgbb.com/)
yea some how I enabled dual iso with cmos [4]
And one last thing the A & B timers make it more stable , seems timer "B" with higher numbers (greater then 1910<) keeps the image in extended resolutions
so it's not only totally depended on just cmos's reg's , timers have a great deal to do with it also .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on June 15, 2018, 07:17:50 AM
I got QEMU up & running finally and i'm looking for some other d4 cam rom's to investigate .
mainly looking for 50D plus T3i (600D) T2i(550d) , T1i (500D)
I have 7D 203 (firmware)   rom . Just PM a link to the roms  , don't post  links here
Thanks. :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: histor on July 06, 2018, 10:04:24 PM
Just a strange bug in the broken3x-12bit build - I've seen it twice with different cards. It destroys the \ML\modules\ folder (or does something wrong to the filesystem). Finally it fails with "Can't find 5D2_212.sym"
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on July 06, 2018, 10:09:26 PM
5D2 handling takes special care. Never! remove card without waiting some seconds for write operations to complete. Yes, cam will access card *after* opening card compartment lid!
Are you sure you took that into consideration?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: histor on July 06, 2018, 10:24:54 PM
The card was in all the time. As well as battery. It's something build specific, I guess.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on July 07, 2018, 07:12:38 AM
Just a strange bug in the broken3x-12bit build - I've seen it twice with different cards. It destroys the \ML\modules\ folder (or does something wrong to the filesystem). Finally it fails with "Can't find 5D2_212.sym"
Thanks for the feed back on the bleeding edge build (not for the light hearted) 
What @Walter said still hold true , even if you open the cf card door to fast while the red light still blinking (writing file for ml) you can get wipe out files or corruption
Yes ,  I've seen this happen before on just the plain nightly builds , but i find it's usually a user error . Pressing  buttons/switch's to fast or too many times , etc. ....
But to really know I would need more info , like all the cam setting , any error messages , etc. ...... , If you clear the setting file in ML on CF card does it still happen ?
If you put the card in your reader , are all the ML files still there ? after ml can't find the .sys file ?
What CF cards are you using ? (needs to be @ least 1000x)

To be clear , this build has limited feature support . That means only basic functions , e.g. all manual adjustment not Auto feature.(frame over ride is OK to use )
Cam must set to manual M mode and Only use pure analog ISO's (100,200,400,800) not digital ISO's (160,320,500,600,etc. ...)
You must turn off(disable) mlv_rec and or mlv_lite before you adjust ISO's or it will crash/lockup .

One more thought , after you get the "Can't find 5D2_212.sym" turn cam off and pull the battery out and try a reboot from cold.
Does it work now ? I've sometimes seen ML miss loading files like "crop marks" or module file , I just pull battery out and then reboot
all is fine usually after that .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: histor on July 07, 2018, 02:03:08 PM
I’ve packed logs and settings. http://disk.karelia.pro/crfYi80/broken3x-12bit-bug.zip
In short - files from the build stay unaltered but all modules vanished. That’s what cart reader sees. While previous crash I saw some alien characters instead of the filenames. Windows can do nothing with them but format.  The second time I’ve just copied the Modules\ back – and everything works with the same settings.
What camera sees when  the error occurs:  the message about 5D2_212.sym and no Global draw overlays. You can shoot, can’t call ML menus. It’s interesting, what was before that, but I can’t roll my mind back.
There was no intensive card writing, the cards are OK (SanDisk Extreme Compact  SDCFXSB-128G-G46 and smaller SDCFXSB-064G-G46). No video recording, only lazy taking pictures in Live view. ISO1000 is a bad setting, but it doesn’t stop taking stills or working live view. Another bad setting is leaving mlv_lite, mlv_rec and mlv_snd enabled together. That’s a cruel way to get 10bit raw with sound. And that’s not a sound of explosion : ) Is there any legal way to do it?
In fact 10-12bit is a fascinating feature for 5d2 and I’m watching  every reddeercity’s  post with great attention. I'm not afraid of multiple crashes now, I have to accommodate with the limitations - to avoid them in the field.
Maybe that’s not very useful information, it’s hard to trace it from actual user error to the final result. My only question is: why ML rewrites modules on shutdown? and what for?
To end with something good – I’m actually charmed with ML for a decade.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on July 07, 2018, 02:22:54 PM
Can you find a way to reproduce the issue? Can you do the same without entering LiveView? Can you narrow it down to some particular module?

Can you get the same issue with regular builds?

ML does not rewrite modules on shutdown; it just saves its settings. However, if there's a bug that causes some sort of memory corruption, its side effects can be just about anything. I'm unable to find other explanation for this behavior.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on July 07, 2018, 02:30:36 PM
In fact 10-12bit is a fascinating feature for 5d2 and I’m watching  every reddeercity’s  post with great attention.

So am I.  I have the 7D and the footage that it provides at 2520x1200 resolution and 10 and 12 bits is absolutely gorgeous.  It has this very exciting film look in it that, in my opinion, only the 7D has.  Don't know about 5DMkII.  I do hope that Reddeercity, with the help of other developers,  succeeds with his most valuable work in porting 2,5, 3 and 4k-crop recording to the 7D.  I keep my thumbs pressed and watch this development with great anxiety!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 12, 2018, 09:48:21 AM
Being spending time working new angles to get UHD cleanly , I thought I would try and reproduce waza57  (https://www.magiclantern.fm/forum/index.php?topic=19336.msg198809#msg198809) experiment in 3x crop mode which he got 2880(V)
-- to date I still can't reproduce his work (I need his variables for reg's 6084 & 6088) . In any case I pushed on ,
I being experimenting with the dm-spy build (has a menu option to select reg's and change them with half shutter button)
-- so find the reg you what to change e.g. (c0f06084 , etc. ....)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fmuotkp%2FVRAM3_small.png&hash=dd2e1282587ae7a5db8e1d4a28c9fa43) (https://imgbb.com/)

Play around with 6084 & 6088 , I noticed that in 3x crop if I change the reg 6084 from 30036(default) to 30037 or 30038 I can chose extended rez's e'g' "3520"
but not 30036 thou the image is distorted at reg 30038
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdzAg5p%2FVRAM6_small.png&hash=3d78b7ca018a94cd07df8818b5079a05)3520x1076 (https://imgbb.com/)

I still can't save a dng/mlv at this point though I can dump the Vram and get images from Liveview.
Things took a turn for the better when decided to check out the reg's in 10x zoom . I noticed that most the reg's are
the same as 3x crop as far as 6084 & 6088 are concerned but
10x reg's from dm-spy
Code: [Select]
[REG] @@@@@@@@@@@@ Start ADTG[1]
[REG] ADTG:[0x105f0301]
[REG] ADTG:[0x10610301]
[REG] ############ Start CMOS
[REG] CMOS:[0x20b]
[REG] CMOS:[0x1e6a]
[REG] CMOS:[0x210e]
[REG] CMOS:[0x3005]
[REG] CMOS:[0x4242]
[REG] CMOS:[0x5c01]
[REG] @@@@@@@@@@@@ Start ADTG[2]
[REG] ADTG:[0x14e00a]
[REG] ADTG:[0x150609]
[REG] @@@@@@@@@@@@ Start ADTG[3]
[REG] ADTG:[0x9011c]
[REG] ADTG:[0xb811c]

3x crop (5x zoom)
Code: [Select]
Start ADTG[1]
[REG] ADTG:[0x105f0301]
[REG] CMOS:[0x20b]
[REG] CMOS:[0x1e6a]
[REG] CMOS:[0x210e]
[REG] CMOS:[0x3005]
[REG] CMOS:[0x4242]
[REG] CMOS:[0x5c01]
Start ADTG[2]
[REG] ADTG:[0x14e00a]
[REG] ADTG:[0x150609]
Start ADTG[3]
[REG] ADTG:[0x93d1a]
[REG] ADTG:[0xbbd1a]

the differences is at "ADTG[3]"
0x9011c & 0xb811c (10x)
0x93d1a & 0xbbd1a (3x)
here the 10x3.5k_dm_spy_dm-0006.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/10x3.5k_dm_spy_dm-0006.log) & 10x3.5k_digic11.LOG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/10x3.5k_digic11.LOG) , why I'm I calling this 3.5k LOG's ?
Will I can chose up to 3520 (V) in MLV_lite  :) right out of the box without any modification's to any reg's and I get a clean Image from vram for Liveview (still no dng , "raw error")
10x zoom mode Liveview dump , I selected 3520 in mlv_lite

Original LV-000.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-000.422) at 3520x1076

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FmnrtJ9%2FLV_000_small.png&hash=483a0e9db141a839d5083c7e743ab312) (https://imgbb.com/)

Original LV-001.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-001.422)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FmnrtJ9%2FLV_000_small.png&hash=483a0e9db141a839d5083c7e743ab312) (https://imgbb.com/)

So I know that 10x zoom is disabled in raw.c if it's enabled can it record the 10x zoom Liveview ?
I think at 10x zoom is really  1:1 pixel , if not how could I select 3520 ? I notice in the dm-log that there is
2 different zoom mode for raw zoom mode "1" (3xcrop) & zoom mode "2" (10x zoom)
part of the log the deal with 10x image process
Code: [Select]
63A82> LiveViewMg:ffa09310:99:02: StartImagePass_x10
63AFC> LiveViewMg:00096224:00:00: *** StartEDmac(0x12, 0x2), from ffa093e8
63B2E> LiveViewMg:00096790:00:00:     addr 1b07800, ptr 1b07800, size
63B9F> LiveViewMg:00096224:00:00: *** ConnectWriteEDmac(0x5, 0x0), from ffa09418
63BFB> LiveViewMg:00096224:00:00: *** RegisterEDmacCompleteCBR(0x5, 0xffa08fa0 "WriteEDmacCrawCompleteCBR_x10", 0x0), from ffa09428
63C56> LiveViewMg:00096224:00:00: *** StartEDmac(0x5, 0x1), from ffa0944c
63C81> LiveViewMg:00096790:00:00:     addr 43e28a4, ptr 43e28a4, size
63CBE> LiveViewMg:ffa0946c:99:02: StartImagePass_x10 CrawAddr : 443e28a4 / KindOfCraw : 0
63D65> LiveViewMg:00096224:00:00: *** StartEDmac(0x8, 0x1), from ffa096b4
63D97> LiveViewMg:00096790:00:00:     addr 44afb4, ptr 44afb4, size
63DD9> LiveViewMg:ffa09df4:99:02: StartIntermediatePassMagnify Addr : 0x50000080
63E3E> LiveViewMg:00096224:00:00: *** StartEDmac(0x0, 0x0), from ffa09e18
63E66> LiveViewMg:00096790:00:00:     addr 10000080, ptr 10000080, size
63ED1> LiveViewMg:00096224:00:00: *** ConnectWriteEDmac(0x11, 0x19), from ffa0b5f4
63F04> LiveViewMg:ffa0b63c:99:02: StartQuarkYuvPass 0x5C578400
63F5C> LiveViewMg:00096224:00:00: *** StartEDmac(0x11, 0x2), from ffa0b660
63F82> LiveViewMg:00096790:00:00:     addr 1c578400, ptr 1c578400, size
6422E> LightMeasu:00096224:00:00: *** SetHPTimerAfter(0x61a8, 0xff86727c, 0xff86727c, 0x0), from ff867310
65BF4> **INT-6Ah*:00096840:00:00: *** TryPostEvent(LiveViewMgr, 0x2, 0x0, 0x0), from ff8d8828
65CBA> **INT-6Dh*:ff9a407c:00:02: [ENG] WriteDMACInterrupt(5)(0x2)(0)
65D01> **INT-6Dh*:00000558:99:02: WriteEDmacCrawCompleteCBR_x10
65D30> LiveViewMg:00096098:00:00: *** LVState: (2) --2--> (3)          ff8e17a8 (x=6bac4c z=0 t=0)
65D73> LiveViewMg:ff8e17c4:98:02: lvVDInterrupt
65DE1> LiveViewMg:00096224:00:00: *** register_interrupt("HEAD3", 0xd9, 0xff986e2c, 0x0), from ff986ee0
65E59> LiveViewMg:00096224:00:00: *** register_interrupt("HEAD4", 0xe0, 0xff986f4c, 0x0), from ff986f98
65EA9> LiveViewMg:ff8da96c:98:02: GetWbIntegFrameForWb 0
65EDD> LiveViewMg:ff8e8fb4:08:01: lvcaeGetWbIntegFrame(sync)
65F02> LiveViewMg:ff8e8fcc:08:01: WBIntegFrame(WB:0x152a40,FST:0x1529dc)
65F79> LiveViewMg:ffa0adfc:99:02: SetWbFstPasParameter
65FAA> LiveViewMg:ffa0b00c:99:02: SetWbIntegParameter
66012> LiveViewMg:ffa06df8:98:02: ReleaseEngineResource Res:4, Free:fffffffc Count:0
66065> LiveViewMg:ff8e1b48:98:02: DispGain:1024, ISO:89
66091> LiveViewMg:ff8e1b70:98:02: WB 1982 1024 1024 2016(4037)
66110> LiveViewMg:ff8e1718:98:02: setHorizontalIrcutData(po0=100)
66198> LiveViewMg:ff863840:82:02: GetVramSize (PUB)
661E8> LiveViewMg:ff867ea8:82:03: SetPBForLV ZoomMode=2, LVType::Info=0, LVType::DispType=0
6621D> LiveViewMg:ff867ed4:82:02: SetPBForLV (3066)
662D1> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
6631E> LiveViewMg:ff8638c4:82:01: ImgDDev SelectParameter DispType=0
66349> LiveViewMg:ff863840:82:02: GetVramSize (PUB)
6637C> LiveViewMg:ff8637e4:82:01: ImgDDev Reg c0f140e0 01b07800
663AE> LiveViewMg:ff8637e4:82:01: ImgDDev Reg c0f140e4 01b07da0
663E1> LiveViewMg:ff86449c:82:02: AsyncEnableImageVBufferForPlayBackAndWait (PUB)
66409> LiveViewMg:ff863fb4:82:01:  RequestNotifyBlank (PRI)


This what I find very interesting
Code: [Select]
661E8> LiveViewMg:ff867ea8:82:03: SetPBForLV ZoomMode=2, LVType::Info=0, LVType::DispType=0
6621D> LiveViewMg:ff867ed4:82:02: SetPBForLV (3066)
ZoomMode=2 so can I assume this is still 14bit raw but out a different channel .
I wonder if SetPBForLV (3066) is the Zoom window for capture ?
as a little side project I'll try to enable 10x zoom recording and see if this work out of the box  :)
That would be nice to just have to push the zoom button twice (10xzoom) to get UHD , this could change everything I've being work on for the last year.
Is this a "secret" special bug that canon dev. forgot to close ?  :o in the same lines as the 50d video that wasn't supposes to be active , here's hoping  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 13, 2018, 08:18:42 AM
Same more info on the above post -- I forgot to add the A & B timers info
Code: [Select]
Desired FPS -------23.976(from30)
Optimize for --------- Low Light
Shutter range --------1/24-1/109
FPS timer A -----------610(FT-28)
FPS timer B -----------1630(FT-10)
Main Clock ------------24.00MHz
Actual FPS ------------24.137

I just check a plain ml build and In 10x zoom you can't select anything over 2k so it must be the FPS timer .
It was a kind of a guess factor , I remembered something that a1ex said about A timers need to be low like (540) can't go below 610
and b timer need to be around 1600-1700 I think , so I play around to get it close and it appears to work in 10x zoom
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FeidCVp%2FVRAM7_small.png&hash=11c2039e748bdf0d5bd7915c4c8a4a80) (https://imgbb.com/)

Also I think 3x crop_mode is a resample of FHD and there's some pixel binding going on , I took 3 screen shots , first FHD then 3x cropmode (5xzoom) and last 10x zoom .
1:1 FHD (line skipping)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FnjUNwU%2FVRAM9_small.png&hash=e046dbec73e62f1286076e459b8ed95f) (https://imgbb.com/)
3x crop_mode (5x zoom)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcNqZAp%2FVRAM10_small.png&hash=60e3dd068050257e42e7dbf68c4aac59) (https://imgbb.com/)
10x zoom
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fc6MSwU%2FVRAM11_small.png&hash=94580b578a7d024c123ff458aa0ac483) (https://imgbb.com/)
I still believe 10x is truly 1:1 of the 5616x3744 with some cropping in movie mode and I think the back lcd sceen is mapped to the sensor in 1:1 so just a 720x480 of the 5.6k image
it I use the digital dolly I can scroll to the top of the image  same as  full res image but with a little cropping maybe 16x9 ? not sure.
I do not get the same results if I use the dolly in 3x crop , I'll post a example of this later on this week .

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 13, 2018, 08:48:42 AM
One more quick little post , while in 10x zoom I did a image dump at 2880 & 3520 (V)
and it seem that the height is being increased in 2880 not sure why ,
maybe the A & B timers are close in tune with 2880 then 3520 (V) .

2880(V)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FbEOXD9%2FLV_003_small.png&hash=896068f762ae708bdbda20fa2338a29a) (https://imgbb.com/)3520(V)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FncAjRU%2FLV_004_small.png&hash=c3e5a7eb6327293b1ab99e041bc04e70) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 15, 2018, 08:27:05 AM
More research in to 10x Zoom to understand if this mode is really 1:1of the sensor or just a resample :
Adjusting the A & B timers are giving my some very interesting results ,
I started with 29 from 30 fps in "low jello" mode , A-timers = 610 (FT-2018) was not changed , I started B-timer at 2333 (FT+ 977)
and increase to 2575 (FT+1219) . In mlv_lite (raw video) set resolution to 3520 (V)

A-timer 610 (FT-2018) B-timer 2333 (FT+977)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fe3K1T9%2FLV_031_small.png&hash=0a04730dfe86c227d2319f1943462810) (https://imgbb.com/)

A-timer610(FT-2018)-B-timer2400(FT+1044)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgnQN1U%2FLV_032_small.png&hash=d1f982559306d1532f0b9736e20d37a0) (https://imgbb.com/)

A-timer610(FT-2018) B-timer2450(FT+1094)  :D
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkVn21U%2FLV_033_small.png&hash=786afbf24c5d2ae1d683f5f0ae95940a) (https://imgbb.com/)

A-timer610(FT-2018) B-timer2500(FT+1144)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FngUt89%2FLV_034_small.png&hash=b428eb4b55268ca0f29f13b02ce1d53c) (https://imgbb.com/)

A-timer610(FT-2018) B-timer2525(FT+1169)  :D
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FiOeoap%2FLV_035_small.png&hash=8284f5c7874fc41e19423c5374801e86) (https://imgbb.com/)

A-timer610(FT-2018) Btimer2550(FT+1194)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FemVGT9%2FLV_037_small.png&hash=ac578c6f761f297c1078d3d9cd0ddaac) (https://imgbb.com/)

A-timer610(FT-2018) B-timer2575(FT+1219)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fc01RBU%2FLV_038_small.png&hash=d10515bc562cf865e9073b5e7d901323) (https://imgbb.com/)

A-timer610(FT-2018) B-timer2581(FT+1225)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhHTCkp%2FLV_039_small.png&hash=315c6bee91307f67db6bc24ede9e4a3b) (https://imgbb.com/)

Now do I have the full res 5.6K image ? I need to apply this with the adtg_gui module so I can adjust the cmos reg's
I still don't know what size the buffer is in 10x Zoom .

This is what the default Liveview Image dump in 10x zoom looks like , I use a plain nightly ML build

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FbOvYQp%2FLV_001_small.png&hash=425aa6f090bbd0f2fa62e7c843383df0) (https://imgbb.com/)

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 16, 2018, 07:58:08 AM
Continue on to research 10x zoom with ADTG_GUI "ISO_Resarch_Branch"
finding the same results with Frame timers A & B as with dm-spy .
Started with 23.976 from 30 fps
default  A & B timers for 10x zoom
Code: [Select]
A-timer 610(-22)
B-timer 1640 (x1.28)

Code: [Select]
A-timer 610(FT-2018)
B-timer 1715 (FT+75)
Here the LV-005.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-005.422) for those timer , getting full res Liveveiw just like the above post .

Loaded the Edmac.mo and it seems that 10x zoom using the same buffer as 3x crop
@a1ex , the math for "B"timer works 1640(base) +75 = 1715 , but for "A" timer 610 seem to be unchanged even though I have (-2018) so with the math that's "-1408" is this being
applied ? do I have to modify them though the reg's (head timers) ? to get 540
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on August 16, 2018, 12:27:31 PM
@a1ex , the math for "B"timer works 1640(base) +75 = 1715 , but for "A" timer 610 seem to be unchanged even though I have (-2018)


That happens when it hits the lower limit for timer A, as declared in fps-engio.c. That is, the value sent to hardware won't be below the declared limit, no matter what you dial in the UI.

To be able to dial a lower value, edit that limit (FPS_TIMER_A_MIN).

If timer A is too small, the side effects begin to be visible on the right side (usually noise instead of image data). If you keep reducing it, the image will break. The ideal value is the smallest value that still gives clean image (with correct image data on the right border).

Quote
I think at 10x zoom is really  1:1 pixel

So is the x5 zoom. The only difference between these modes is how the preview is rendered. In both x5 and x10 modes, the captured raw image is identical. So, I see no reason why anything would work better in x10, rather than in x5.

For the UI, I prefer to leave the x10 mode untouched, for focusing.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 19, 2018, 08:48:39 AM
Will a little good & bad news , first the good -- I figured out what reg's & timers I need for extended resolutions as seen in the previous post's with the Liveview_dumps I've posted .
Now the bad -- the problem I'm having is I'm lock in to the frame buffer (LiveviewBuffer) of either 1880x1248 or 2144x1076 (3x crop_mode) I can't extent the buffer ,
so I'm getting a doubling of the same frame buffer as seen in 4096x1248 here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg201004#msg201004) & here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg201033#msg201033) . I still can't extend the 3x crop window to 3k like @waza57 did (If you are out there waza57 please can you comment on your experiment ) But I have been able to extent 1:1 (FHD) to 2144x1076 from 1880x1248 . Thou it uses the frame resolutions of 2.1k it's still has the 1880x1248 frame buffer , it just overlaps part the same frame buffer to fill the frame resolution , I guess that has something to do with the edmac .

Liveview_dump 422 , frame res. is 2144 (V) in 1:1 FHD and is filled with the 1856 (V) Liveview buffer , see the black bar then part of the next same frame
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FbEzKqz%2FLV_022_small.png&hash=f69309d4d81044bedb1fb36d82486a23) (https://imgbb.com/)

I guess I I'm stuck now at this point , been going around in circles and unless I can figure out the liveview buffer (This may be beyond my ability  :-[ )
a1ex is there any hint for this ? I'm lost on the Liveview buffer , or I'm I missing a step . I would think I should be able to save single dng even in extended res.
 
For the UI, I prefer to leave the x10 mode untouched, for focusing.
Ok , just looking to see if there is any usable information that can be used
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 20, 2018, 09:30:18 AM
Switch back to "ISO_Research_Branch" getting much better results , thou the buffer is still a problem.
Finding "CMOS[0]" can change the Liveview to full width of the sensor from a center crop.
Here are the setting I found to be the most stable and cleanest image while there's still lots of room for improvement .

What I did to get the results that I'm posting -- Load "iso research branch" build (I added 4096 (V) & 3840 (V) to mlv_lite.c  for extended resolution .
Loaded
Code: [Select]
adtg_gui.mo
crop_rec.mo
edmac.mo **(only for info)**
mlv_lite.mo
enabled "crop_rec.mo" , set to 1:1
enabled  "adtg_gui.mo" , toggled on "digic reg's" then enabled the 3K 5d3 preset(I changed the reg's to work with 5d2 ,6808-->6088 , 6804 -->6084)
that it , and trial & error got me to this point .
Code: [Select]
ADTG[0x1061] 0x249 -->3ca      **(1:1 frame Blanking)**
CMOS[0] 0x20b -->0xe1b
CMOS[1] 0xc00 -->0x10
CMOS[2] 0x40e -->0x10c

Here's the Liveview image dump for those setting -- frame resolution set to 4096x1250

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcLb3kz%2FLV_031_small.png&hash=420c00a3426776e12f4b532ccf2689f1) (https://imgbb.com/)

And the HD image dump for the above Liveview dump

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fi6dfyK%2FHD_031_small.png&hash=17a772dfa4bc23c103d3924861e875a9) (https://imgbb.com/)


when CMOS[0] is disabled (no over riding reg's) the image turn back to center crop .
Code: [Select]
ADTG[0x1061] 0x249 -->3ca      **(1:1 frame Blanking)**
CMOS[1] 0xc00 -->0x10
CMOS[2] 0x40e -->0x10c

Liveview image dump -- frame resolution set to 4096x1250

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhhmEse%2FLV_029_small.png&hash=d99b6518431c414bfbec0191204a52de) (https://imgbb.com/)

HD image dump -- frame resolution as set to 4096x1250

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FeJdYkz%2FHD_029_small.png&hash=0c27b58645506a79e1536f9a6d66509e) (https://imgbb.com/)

So any one with a d4 (digic4) cam 7d , 50d etc. ... can follow along -- the process should be very similar

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 25, 2018, 09:41:13 AM
I'm starting to smell success !
Made go great process today with "crop_rec_4k_5D2" branch from "wasa57" in 3x crop_mode
I was able to get the vertical to 2304 (V) , max is 2144(V) thou I couldn't save a dng at that rez ,
but I was about to save a 2252x1075 dng after ml crop it , so really it's 2416x1127 the un-crop size or the "Raw Size"
Code: [Select]
Active Area  : 52 160 1127 2416

Here the Original dng from the Image_dump 2256x1075-RAW-000.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2256x1075-RAW-000.DNG)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fh7xVA9%2F2256x1075_RAW_000_small.png&hash=6b50ce5443207d818ddb6eacdecaad4a) (https://imgbb.com/)

here is also 2256x1075-RAW-000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2256x1075-RAW-000.png) from the same image dump

 
Code: [Select]
ExifTool Version Number         : 10.45
File Name                       : RAW-000.DNG
Directory                       : C:/dcraw
File Size                       : 4.6 MB
File Modification Date/Time     : 2018:08:24 22:18:22-06:00
File Access Date/Time           : 2018:08:11 23:06:24-06:00
File Creation Date/Time         : 2018:08:11 23:06:24-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               :
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Modify Date                     :
Artist                          :
Subfile Type                    : Full-resolution Image
Image Width                     : 2416
Image Height                    : 1127
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1127
Strip Byte Counts               : 4764956
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 1278
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 2256 1075
Active Area                     : 52 160 1127 2416
Opcode List 1                   : (Binary data 28 bytes, use -b option to extrac
t)
Copyright                       :
Exposure Time                   : 0
F Number                        : 0
Exposure Program                : Not Defined
ISO                             : 0
Exif Version                    : 0221
Date/Time Original              :
Exposure Compensation           : 0
Max Aperture Value              : 1.0
Metering Mode                   : Unknown
Flash                           : No Flash
Focal Length                    : 0.0 mm
Sub Sec Time                    :
Sub Sec Time Original           :
Focal Length In 35mm Format     : 0 mm
TIFF-EP Standard ID             : 1 0 0 0
Serial Number                   :
Lens Model                      :
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 2416x1127
Megapixels                      : 2.7
Shutter Speed                   : 0

Just adjusting with digic poke in the debug menu ,
 
Code: [Select]
c0f06084 - 0x30002
c0f06088 - 0x469ff77
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: honza123 on August 25, 2018, 10:28:30 AM
Great !
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 29, 2018, 05:51:05 AM
Now that a1ex fixed adtg_gui.c in iso-resaerch branch I can proceed to get the full raw buffer for full raw width
Here's (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/) (magiclantern-iso-research-4k-D4.2018Aug29.5D2212.zip) the Up dated build I'm using , did some quick adjustments to 6084 & 6088 . Got a error printed on the screen

Code: [Select]
Default raw buffer too small (38MB , need 2148x16377 59MB - reallocating
[BKT] giving up
black 2/5 stdev too large (47077/100 ref 15535/100

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fd6n1a9%2FIMG_2620_small.png&hash=b838cab550734bdea49075ccfa73083d) (https://imgbb.com/)

Will at lest I know I'm working off a 38MB buffer -- do I need a 59MB buffer ?
@a1ex , what would  the buffer size the 5D3 uses for 4k extended res ? does it need to be 59MB

Here are the 6084 & 6088 settings , I first enable the digic reg's , exited to liveview press the play button reviewed a cr2 photo
re-entered adtg_gui.mo enabled the 3k preset for crop rec -- adjusted reg's 6084 & 6088

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fd6KToU%2FIMG_2625_small.png&hash=a9d34ee3dc6d14900130ea31cbd9c1c0) (https://imgbb.com/)

Shot a short 25 second 720p video on my iPhone , showing the error being printed on the screen with some other info
https://drive.google.com/file/d/129ONClwQUoZrQeJ9tg0moK7YnIeRdzKm/view?usp=sharing

the setting for 6088 was just a random number to see if the changes do really take effect , and they do

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on August 30, 2018, 07:55:13 AM
59MB is way more than what you can allocate contiguously on these cameras; that number comes from the value dialed to C0F06088:

0x3FF90433 -> split in two -> 0x3FF9 and 0x433.
0x1 -> split in two -> 0x0 and 0x1 (better don't touch this one; leave it at 0x10036).
Difference: 0x3FF8 and 0x3FD.
The latter is X resolution divided by 2 (number of columns read out at once). That's how you've got 2148.
The former is Y resolution, but higher bits might be unused (hardware does not interpreted them, i.e. it works in the same way whether these bits are 1 or 0). However, ML interprets them.
So, 0x3FF8 = 16376. ML requests memory for 2148x16377 (looks like in your source, it adds 1). The hardware delivers 0xFF8 (4088) or 0x7F8 (2040); likely the former.

In your previous example, 0x469 - 3 = 1126; not sure how to interpret 0xff77 - 2 to get close to 2256...

In any case, I suggest adding small quantities to either the low halfword (for X) or to high one (for Y).

As you increase X, you may have to increase FPS timer A by the same amount (1 step = 2 pixels on 5D2), and possibly other registers that use similar values. For example, on 700D, timer A must be written to 0xC0F06008, 0xC0F0600C, 0xC0F06010, 0xC0F06824, 0xC0F06828, 0xC0F0682C and 0xC0F06830; the last 4 must be in sync and if you don't increment them, you will get just black bars.

As you increase Y, you may have to increase FPS timer B and PowerSaveTiming registers (https://www.magiclantern.fm/forum/index.php?topic=19300.msg202783#msg202783).

Other registers can be figured out by comparing x5 (1:1 crop) with plain 1080p (that's pretty much all you've got on 5D2). On 700D, comparing the regular 1080p with the 1:1 crop mode from Canon firmware was very helpful.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 30, 2018, 08:06:51 AM
Thanks @a1ex that's exactly what I was looking for , makes sense now .
I was blinding adjusting , now I have more organized approached thanks.
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 30, 2018, 08:41:27 AM
From 6088 "default" 4f40432 in 1:1 FHD
4f4 =1268 (V or X)  432 = 1074 x 2 = 2148 (H or Y)
I tried 5f40432 to see what happens
5f4 = 1524 (V)  432 = 1074 x 2 =2148 (H)
increased vertically by 256 pixels if I did the math right .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on August 31, 2018, 08:47:07 AM
Looking up now ,  :)
I'm taken baby steps , able to increase the height now with
c0f06088 , A & B timers , target height is 2160 -- I'm at 1760
Code: [Select]
cmos[2]  40E --> 10E
Code: [Select]
c0f06088
4f40432 ->6f40432

Code: [Select]
Low Jello , 180d
A timers  --> 792
B timer --> 1750 (around there)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjEuRf9%2FVRAM2_small_6088.png&hash=13b45b8ab1c81b852eaa9674dea0cc0c) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fd9QwDU%2FVRAM1_small.png&hash=88155dadadf47ac6ff2cc5c891713e15) (https://imgbb.com/)

disregard the 3200 , I haven't adjust the lower half yet (432) i did quickly set the lower half to 632 to the first half of 6f4 and got it to lock to 2880x1760 & 3072x1760
but it lock up , needs more reg's to make it stable , I'm slowly finding the reg that correspond to the 5d3/D5 camera.
Can't save a dng yet , thou liveview dumps look promising .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on August 31, 2018, 07:48:13 PM
Made a couple experiments on 5D2, too. Initial notes:

- black level depends on raw type (either 1024 or 1792; need to sit down to find it)
- pink preview still stubborn
- CONFIG_EDMAC_RAW_SLURP kinda works, but gives corrupted frames
- timing for raw_lv_vsync appears to be correct
- calling raw_lv_vsync from the same place where Canon code would call it (StartImagePass x1/x5 CrawAddr / KindOfCraw), did not help
- patching the SetEDmac call from the above routine appears to work:

Code: [Select]
static void raw_lv_setedmac_patch(uint32_t* regs, uint32_t* stack, uint32_t pc)
{
    /* R0: EDMAC channel */
    /* R1: output buffer */
    /* R2: EDMAC info (geometry) */
    /* R3: flags */

    int width, height;
    int ok = raw_lv_get_resolution(&width, &height);
    if (ok)
    {
        if (redirected_raw_buffer)
        {
            /* optionally update output buffer */
            regs[1] = redirected_raw_buffer;
        }

        /* update EDMAC image size */
        int pitch = width * raw_info.bits_per_pixel / 8;
        static struct edmac_info dst_edmac_info;
        dst_edmac_info.xb = pitch;
        dst_edmac_info.yb = height - 1;
        regs[2] = (uint32_t) &dst_edmac_info;

        /* we can override this here */
        EngDrvOut(RAW_TYPE_REGISTER, lv_raw_type);
    }
}

static void raw_init()
{
    ...

    /* SetEDmac from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
    /* FIXME: why it fails when the hook is placed on the BL instruction?! */
    patch_hook_function(0xFFA08B20, MEM(0xFFA08B20), raw_lv_setedmac_patch, "RAW LV x5");
    patch_hook_function(0xFFA08008, MEM(0xFFA08008), raw_lv_setedmac_patch, "RAW LV x1");
}

Still need to figure out why our code called from the same place doesn't work well, but the patched one does.

Now, adtg_gui stuff. FPS timer A has a small quirk - one unit means 4 pixels on this camera, but one unit of C0F06088 means 2 pixels. So, the easiest experiment is:
- enable ADTG registers and (in the advanced tab) ENGIO registers
- go to x5 zoom, enable mlv_rec's grayscale preview
- override FPS timer A (C0F06008) from 0x2770277 to 0x2770377 (increase X by 0x100)
- this change lets you increase raw resolution register (C0F06088) by 0x200, i.e. from 0x46A04BA to 0x46A06BA
- this gives 3168 pixels horizontally in mlv_rec: (0x6BA - 0x36)*2 = 3336, minus 160px black bar = 3176, then 3168 is multiple of 32.

Next is adjusting CMOS registers for centering the image horizontally - reply #10. Will edit later.



Part 2. Let's try to get full width LiveView.

Target values: the ones from photo mode.
Code: [Select]
              x1     x5 centered        photo    description
C0F06084:   0x10036     0x30036        0x10037   start row/column
C0F06088: 0x4F40432   0x46A04BA      0xEDD0B87   stop row/column (column: 1 unit = 2 pixels)
C0F06008: 0x23B023B   0x2770277      0x5DB05DB   FPS timer A (1 unit = 4 pixels horizontally)
C0F06014:     0x6D5       0x4FF          0xEDC   FPS timer B (1 unit = 1 pixel vertically)
CMOS[1] :     0xC00       0xE6A          0xC00   vertical start/stop
CMOS[2] :     0x40E       0x10E            0x8   horizontal pos & binning mode
CMOS[3] :       0x5         0x5            0x7   ???
CMOS[4] :     0x244       0x244          0x244   ???
CMOS[5] :       0x1        0x1           0xC05   ???

Code: [Select]
CMOS[2]:
xx-- -------- binning mode (00: read every column, 01: bin every 3 columns, 10 and 11: bin even more columns?!)
--xx xxxx---- horizontal position (0: start from left, 0x2a: thin stripe of valid data, 0x2b: no more valid data)
---- ----xxxx ??? (0x8 in photo mode, 0xE in LiveView, double highlights with 0x8 in LV)

For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.



Now, to increase height.
C0F06014: 0xEDC (copied from photo mode) => 4.2 FPS, clean image, so far so good.
C0F06088: 0x56A0B87 (increase Y by 0x100). Image breaks. Undo the change and look for other registers that might be related to timer B (close to 0x4FF)
First candidates: HEAD timers. Attempting to change them results in camera lockup. Back to the drawing board.

Some success starting from vanilla x5 and adjusting just the height:
- change C0F07150 from 0x49C to 0x59C and pray it won't lock up
- change C0F0713C from 0x476 to 0x576 and pray it won't lock up
- change C0F06088 from 0x46A04BA to 0x56A04BA => mlv_rec shows 2144x1330, but image is not good on the bottom half
- try to change CMOS[1] => E69 looks better, but image stutters; F6A looks even better, also with stutter, still some image data missing
- look for powersave timing registers: ADTG1[1047] from 0x550 to 0x650 appears to do the trick. maybe not needed? go to 1x and back to x5 to refresh.
=> 2144x1330 in mlv_rec with clean preview, still with some stutter with ML grayscale preview, but looks fine with Canon preview.

Let's try some more:
- C0F07150: 0x69C
- C0F0713C: 0x676
- C0F06088 to 0x66A04BA
=> 2144x1586 apparently working just fine.

- C0F07150: 0x79C
- C0F0713C: 0x776
- C0F06088 to 0x76A04BA
=> 2144x1842 apparently working just fine.

- CMOS[1]: 0xC00
- C0F07150: 0x79C
- C0F0713C: 0x776
- C0F06088 to 0x76A04BA
=> 2144x2354 apparently working just fine.

Battery empty, will continue tomorrow.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 01, 2018, 04:42:34 AM
Holy Cow !!!  :o ,  it's not my birthday & I know it's not 'Xmas , just look outside to make sure it wasn't snowing  :D
5D2 & D4 users  a1ex has given use a amazing gift ! the proof that 4k is a real possibility on 5d2 or at least 3k  8)

This is incredible , thanks @a1ex -- so I guess I should try and reproduce some of your experiments (by the way I like your experiments way better then mind)
I knew about these one 's but anyway lock up cam
Code: [Select]
C0F07150:
C0F0713C
"CONFIG_EDMAC_RAW_SLURP" sound very encouraging

Anyhow here we go , let see if I can save a good dng in extended rez.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 01, 2018, 06:22:20 AM
Quote
easiest experiment
Ok I can reproduce , got even an dng to export thought the image dump -- thou it's all distorted

Code: [Select]
Default Crop Size               : 3176 1075
Active Area                     : 52 160 1127 3336

dng from image dump converted to png so I can post it
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdapqAK%2FRAW_001_small.png&hash=991d003faa821a90e198b066c3e1aa10) (https://imgbb.com/)

Liveview .422 image
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkJ8xPe%2FLV_007_small.png&hash=635053d107276c29e2452836a3d06e26) (https://imgbb.com/)

Here the original 3176x1075 RAW-001.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-001.DNG) & LV-007.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-007.422) image .

Code: [Select]
ExifTool Version Number         : 10.45
File Name                       : RAW-001.DNG
Directory                       : C:/dcraw
File Size                       : 6.3 MB
File Modification Date/Time     : 2018:08:31 21:37:14-06:00
File Access Date/Time           : 2018:08:31 21:52:05-06:00
File Creation Date/Time         : 2018:08:31 21:52:05-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Subfile Type                    : Full-resolution Image
Image Width                     : 3336
Image Height                    : 1127
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1127
Strip Byte Counts               : 6579426
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 8833
Default Crop Size               : 3176 1075
Active Area                     : 52 160 1127 3336

White Level                     : 16200
Default Crop Origin             : 0 0
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 3336x1127
Megapixels                      : 3.8

On to the next experiment .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 01, 2018, 08:15:24 AM
Got to full width "5632x1075" for a short time then lock up , but I was able to save a image dump only good thing I got was a Liveview .422
scrambled up image , I didn't get a chance to change 6008 before it locked up .

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FircMZe%2FLV_008_small.png&hash=167f3cc87f25c3c1df84d36b409a2aa0) (https://imgbb.com/)

Here the LV-008.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-008.422) from the rez of 5632x1074

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 01, 2018, 11:51:04 AM
Hm, the DNGs should be clean, not sure what's missing. Did you start overriding from x5 zoom?

Anyway, back to full height.

CMOS[1]: 0xC00
C0F06014: 0xEDC (increased by 0x9DD)
C0F07150: need to increase by 0x9DD from 0x49C to 0xE79.
C0F0713C: need to increase by 0x9DD from 0x476 to 0xE5E.
C0F06088: 0x46a04ba + 0x9DD0000 = 0xE4704BA (a little lower than full height)
=> 2144x3600 in mlv_rec; the DNG from VRAM dumps looks clean.

Code: [Select]
Default Crop Size               : 2152 3600
Active Area                     : 52 160 3652 2312

Still need to reach 0xEDD04BA for full height (150 pixels left). Dialing this value doesn't work.

Let's increase everything by 150 units (0x9DD + 150 = 0xA73).

CMOS[1]: 0xC00
C0F06014: 0xF72
C0F07150: 0xF0F
C0F0713C: 0xEE9
C0F06088: 0xEDD04BA
=> 2144x3750 in mlv_rec, 9.6 FPS.

Let's add the full width:

CMOS[2] = 0x00E
C0F06008: 0x27705DB
C0F06084: 0x10037 (same as photo mode)
C0F06088: 0xEDD0B87 (same as photo mode)
=> 5632x3752 in mlv_rec, 4 FPS, DNG looks clean to me.
Code: [Select]
Default Crop Size               : 5632 3752
Active Area                     : 52 160 3804 5792

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2F5D2%2Ffullres-lv.png&hash=92c3481efd766edf41af7edd9e26bcd4)

May require some fine-tuning to match a CR2, maybe the frame rate can be slightly increased, but this completes the proof of concept: arbitrary resolutions, up to full-res LiveView, are possible on 5D2 and likely on all other models already running ML.

This experiment was actually a continuation of the 700D full-res LV PoC (https://www.magiclantern.fm/forum/index.php?topic=19300.msg205546#msg205546).

Will commit the changes and update the experimental builds to make sure the above can be reproduced by any of you.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 01, 2018, 01:49:44 PM
I have the same problem with C0F07150 and C0F0713C I can't override them correctly in my implementation of a 
crop_rec module for 5D2. Maybe that's why I'm limited in height

A few month ago i posted a dng with a promising size from my experiences in "dm-spy-experiments" branch.
It took a lot of time to implement this method in a module of crop_rec_4k branch.
I've seen that Reddeercity spend a lot of time time here, it's encouraged me to continue
 
for the moment only "CROP_PRESET_3X" and "CROP_PRESET_CENTER_Z".For those who want to test the code is there:

https://bitbucket.org/waza57/magic-lantern/src/crop_rec_4k_5D2/ (https://bitbucket.org/waza57/magic-lantern/src/crop_rec_4k_5D2/)
Download link:
magiclantern-Nightly.2018Sep02.5D2212.zip (https://bitbucket.org/waza57/magic-lantern/downloads/)
 
I hope it will be useful. It works for me but at your own risk (Digic poke is enable don't use it if you do not know what you are doing)

SETTINGS:
   
      Canon menu : NTSC 30
      ML menu    : set resolution to maximum ,  preview grayscal
      Modules to load: crop-rec, mlv play, mlv rec, silent
   
NOTES for developers:

I don't use CONFIG_EDMAC_RAW_SLURP.
 
I started from this:
 https://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909 (https://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909)
 
https://www.magiclantern.fm/forum/index.php?topic=10111.msg125581#msg125581 (https://www.magiclantern.fm/forum/index.php?topic=10111.msg125581#msg125581)
 
And especially this where i decide to try increment height with patching ROM addresses.
 https://www.magiclantern.fm/forum/index.php?topic=19336.msg183200#msg183200 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg183200#msg183200)
 
Thanks to GREG!
 
For crop_rec.mo works i need to make change in functions of:

 raw.c

          "raw_lv_get_resolution" ->  ...uint32_t top_left  = shamem_read(0xC0F06800)... to retrieve the size like this
          "autodetect_black_level"->  disabled,  otherwise  i got "raw detect error"
          "#define BLACK_LEVEL 1792"  otherwise bad black level
         
edmac-memcpy.c , fio-ml.c
          I need to disable call to function "clean_d_cache()" otherwise i can't patch ROM addresses
 
lv-img-engio.c
          I need to add this otherwise impossible to compile : "if defined(FEATURE_EXPO_ISO_DIGIC) && defined(CONFIG_EDMAC_RAW_SLURP)"
         
Thanks to a1ex for this project, without him, I would never have discovered the joys of programming in C.

TODO list:

- Increase height frame.
- mlv_lite module compilation.
- Compressed raw activation. ( new idea here too :Danne Idea  (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205681#msg205681))
- Shutter speed question ( is this bad?)
- compatibility with Dual_iso module question.
- ISO adjustment activation (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205719#msg205719 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205719#msg205719))
- Faulty HDMI connection
- Bad preview solution

* And all i will notice in your feedback
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 01, 2018, 03:45:07 PM
Right, it took a lot of trial and error to figure out the meaning of all these registers. Back then, when I started to write the initial 4K hack, I didn't understand how this works well enough to make a generic implementation; that's why crop_rec ended up preset-based, with each resolution being manually tuned.

Now looking into 500D. This one is interesting because its 1080p mode is limited to 20 FPS. It does have 720p30 though. Let's compare them:

Code: [Select]
          1080p20  720p30  x5 zoom    photo
C0F06008  50F050F 5630563  58F058F  AF90AF9
C0F06014      4D1     304      3FF      C83
C0F06084    20040   20040    2004A    1004C
C0F06088  43D06C0 29506C0  360085A  C84132C
C0F0713C      440     294      370   unused
C0F07150      46E     2A1      388   unused

First mystery is at timer A: both 1080p20 and 720p30 use the same horizontal resolution, yet there are different values of timer A. Dialing down the value from 720p (0x563) to 0x50F does not work; image breaks below 0x543. Why?!

1080p: 1664x1083
720p: 1664x659

It's definitely one of the ADTG registers. Locked all of them to their values from 1080p, switched to 720p et voilà! 32 FPS with timer A = 50F!

Narrowed down to ADTG[100C] - the line skipping factor. Boo...



Full width (from x5):

FPS timer A: 1 unit = 2 pixels.
C0F06088: 1 unit = 1 pixel.
2 * 0x58F - 0x85A = 710
2 * 0xAF9 - 0x132c = 708
=>
C0F06008: 58F0AF9
C0F06084: 2004C
C0F06088: 360132C

Appears to work, but need to implement the display filters first in order to check the results. Will revisit later.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 02, 2018, 06:59:51 AM
Ok starting over , trying the first experiment again to get 3k (H) in 3x crop
Still can't get a clean dng out , even try a fresh iso-reserach branch clone no go , there must be something I'm missing!
I can't get it to work at all with mlv_rec , kind of works with mlv_lite  -- so a1ex are you using mlv_lite?
even with mlv_lite I'm still at max rez of 2144 even if I try the full width , I get the res in the menu 5623 but still say max 2144.

It work better with my very hacky "raw_slurp" with the default raw at 4096X4096 (the one I post come from that )

Do I need to add that patch code you posted ? to made it work so I can export clean dng's ?

I followed your procedure
Code: [Select]
- enable ADTG registers and (in the advanced tab) ENGIO registers
- go to x5 zoom, enable mlv_rec's grayscale preview
- override FPS timer A (C0F06008) from 0x2770277 to 0x2770377 (increase X by 0x100)
- this change lets you increase raw resolution register (C0F06088) by 0x200, i.e. from 0x46A04BA to 0x46A06BA
- this gives 3168 pixels horizontally in mlv_rec: (0x6BA - 0x36)*2 = 3336, minus 160px black bar = 3176, then 3168 is multiple of 32.

When I got it to show the right size 3168x1074 it was with mlv_lite & "raw_slrup" with default raw buffer 4096x4096 which I thing is 32MB

I also had Frame Ride disabled , and I did see with the full width 5632 It was at 12.5 fps but image was garbage .

Should I try the 4k crop record branch with ADTG_GUI.mo ?
This confusing me why I can't it to work , the image is always garbage/noise/distorted .

@waza57 , good to see you back here !
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 02, 2018, 07:12:21 AM
....some unnecessary deleted modes at the moment in the crop_rec module :

new link : magiclantern-Nightly.2018Sep02.5D2212.zip (https://bitbucket.org/waza57/magic-lantern/downloads/)

@ a1ex
Do you see a ADTG[100C] with adtg_ gui ? i'don't see it with 5D2.

@reddeercity
i have tried to simply copy the files from the adtg_gui folder of iso-research into the crop_rec_4k branch. it seems to work.


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 02, 2018, 08:36:14 AM
Great job @waza57 ! Tried you crop_record 4k build , and I can confirm 3xcrop 3.5k preset works @ 2880x1080 23.976 10bit out of the box !

Liveview image dump is good LV-011.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-011.422)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkKBUNz%2FLV_011_small.png&hash=1a85027741d484fecce822e5451ae9c6) (https://imgbb.com/)

Image dump dng also good RAW-003.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-003.DNG)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FddR79e%2FRAW_003_3k_small.png&hash=0f9fb545043a174d22b84b26cc56a0d5) (https://imgbb.com/)

Code: [Select]
Default Crop Size               : 2884 1080
Active Area                     : 54 264 1134 3148
Black Level                     : 1792
White Level                     : 16200

Code: [Select]
File Size                       : 6.4 MB
File Modification Date/Time     : 2018:09:01 23:34:10-06:00
File Access Date/Time           : 2018:09:01 23:58:53-06:00
File Creation Date/Time         : 2018:09:01 23:58:53-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Subfile Type                    : Full-resolution Image
Image Width                     : 3192
Image Height                    : 1200
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1200
Strip Byte Counts               : 6703200
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 1792
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 2884 1080
Active Area                     : 54 264 1134 3148
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 3192x1200
Megapixels                      : 3.8

Recorded a short 12 seconds raw video @10bit 2880x1080 23.976 here the ProRes_2880x1080_24p (https://drive.google.com/file/d/1Yae_P2I-ORNUv3xuDENs6r3iqm9y5dV-/view?usp=sharing) version of it , around 600 MB
The original raw video M01-2333.MLV (https://drive.google.com/file/d/1FTtNPhspYlu8HQgrWGIZi9qdbwujwRoe/view?usp=sharing) , it's about a 1GB
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fg3v9pe%2Fmlvproduder_2880x1080_5d2_10bit.png&hash=7cd7b04c874962b2c209bcae6555c8a2) (https://imgbb.com/)

Here a frame from that
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fb7dqvK%2FM01_2333_000113_small.png&hash=13dd2b3139e31d0cb69d73d187f0aa93) (https://imgbb.com/)

Awesome job waza57 , works great so far .
I must of really messed up the raw.c somewhere in my builds , I have to check out your code and see where I when wrong .
Thanks again  :D

If anyone missed it 5D2 10 bit 3K raw video is a reality !
Another Pipe Dream comes true  8)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Ottoga on September 02, 2018, 09:10:34 AM
Quote
If anyone missed it 5D2 10 bit 3K raw video is a reality !
Another Pipe Dream comes true  8)

So cool Reddeercity, persistence and perseverance really paid off.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: scotophorus on September 02, 2018, 09:16:06 AM
i've been reading your updates on this thread for the last year and a half and now i have to spam you some thanks for this great achiement: holy sh*t! that's amazing! thank you very much @alex @reddercity @waza57 @dfort and everyone involved
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 02, 2018, 10:09:25 AM
@ a1ex
Do you see a ADTG[100C] with adtg_ gui ? i'don't see it with 5D2.

Yes, I see it after getting out of LiveView and back. Works as expected.

Looking for a 720p mode on 5D2? Now we need to figure out how to reduce the resolution in order to increase the frame rate :)

On 700D I've ran into a restriction that appears to be related to the live preview: modified resolution could not have an area (WxH) smaller than in the original video mode. The behavior is likely there on earlier models as well. What I think it's going on: the image processing modules that create the preview image will not get enough input data, and they will just keep waiting, freezing the preview and eventually crashing the camera. If the modified resolution is larger, the modules get enough input data, possibly with incorrect dimensions, they discard the extra data and the camera ends up displaying something (a garbled image); and it doesn't lock up.

edit: reducing resolution seems to work, to some extent, LiveView is frozen, camera didn't crash after ~ 1 minute (didn't try more), there are some black level issues, mlv_rec shows 1856x704 at 50.004 FPS.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2F5D2%2F720p.png&hash=61889ee14a6c323ad368cdc2b384c1c2)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: kwstas on September 02, 2018, 11:31:15 AM
it's alive! congrats guys!

I can confirm
20 sec mlv, 2880x1080 @ 10 bit, though a -116px black at the right
13 sec mlv, 2540x1080 @ 12 bit

my card does 77mb at average

--
edit: cropped mode a bit unstable yet, seems not to be working now
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 02, 2018, 02:42:24 PM
@ a1ex

OK, thanks. I have clone  crop-rec4k branch and work on it with a copy of the adtg_gui folder of iso-research branch.
The things compile and seems to work like this and permitted a minimal  crop_work module to work, but:

adtg[100c] is not present even if i refresh livevew or go to play mode and back or go to x5 an go to play mode and...and.. ...nothing ...nothing.

C0F0713C  and C0F07150  i pray for  it won't lock up but god was not in my build.....

it's why , i would like to know, if it's possible:
 
1 - what is exactly the reference of your commit.
2 - Also, if you are currently using your "raw_lv_setedmac_patch" solution?
3 - if yes , is this solution works only with CONFIG_EDMAC_RAW_SLURP
4- And because I'm a little lost , what this solution does concretely.

@reddeercity

happy birthday anyway!

@kwstas

Do you use the new download link ? what is the problem ? have you apply my recommanded settings ?

 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: kwstas on September 02, 2018, 06:33:18 PM
@waza57

fixed, the problem occurs while FPS override is enabled.
Yet, 1920 1:1 does not work for me and 3.5k gives me a freezed display while recording, but I think that happened before.
Basically, I wish we don't have to zoom in 5x mode and have all this side effects (low fps, greyscale, freezes etc) and use just some basic crop rectangle in 1:1 instead.. but I guess there should be some limitations.

Nevertheless superb job considering this September mkII celebrates its 10-year birthday.. :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 02, 2018, 08:01:23 PM
@ kwstas

Thank you for the feedback.
I'm not sure to correctly understand but :

Quote
the problem occurs while FPS override is enabled
Yes I have not alerted about this because I think it is the case for all cameras model with crop_rec module . if someone can confirm ?

Quote
freezed display while recording
the preview is bad when recording but the recording is good , no ? same thing ,  if someone can confirm wit other cameras model ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: kwstas on September 02, 2018, 08:30:27 PM
the preview is bad when recording but the recording is good , no ? same thing ,  if someone can confirm wit other cameras model ?

exactly, recording is flawless, preview is freezed.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 03, 2018, 06:01:43 AM
The frozen Liveview did happen on the 5D3 in crop_rec 4K if memory serve me correctly ,
I do think a1ex fixed that in mlv_lite , that the one I've been working from .
if you can fix the compiling problem/error in mlv_lite , has something to do with compressed raw 10,12 bit ?
I think I had it compiling in d4 4k crop_rec  , I do believe I just commented out the 10-12bit compressed stuff .
With mlv_lite the preview should be better , it where the bulk of the crop_rec 4k stuff is .

I can get around 30 second (average 79-80MB/s) after playing around with the srm write mode , "4" seem to be the fastest until a frame skips .
I did enabled "frame skipping" and shot a 8GB 1:40 second 2880x1080 clip , I really did notice any missing frames (unless it was just 1 frame)
It look seamless after rendering to a prores , next I'll try a talking head and see if there's a noticeable gap -- even if there is run a "bcam"
and kept recording 3k , if it's just one or two frame I can live with that for the monument .

Now next we need compression , that's lossless compressed raw . We are very close on the that front I think , the problem is the cam lockup's upon
saving a lossless silent photo (when we can save a lossless silent photo we should have it for raw video)
Here where we are at right now with 5D2/D4 cam's ProcessTwoInTwoOutLosslessPath  (https://www.magiclantern.fm/forum/index.php?topic=18443.msg205333#msg205333)

There may be another way also to increase the CF card write speed --
I notice in one of my LOG's that for some reason 5d2 clocks the UDMA at 6 , where the cards are rated for UDMA 7
Code: [Select]
825FE>  CSMgrTask:00095f98:00:00: *** register_interrupt("CFDriver", 0x82, 0xffb8b8cc, 0x0), from ffb8bb58
      .............
834A0>  CSMgrTask:ffb8a92c:22:05: [ID:Model Number] = LEXAR ATA FLASH CARD                   
834C8>  CSMgrTask:ffb8aabc:22:05: IDE = 4, PCMCIA = 80, UDMA = 6
      .............
83572>  CSMgrTask:ffbdb8a0:22:01: cfDecideTiming: UDMA Mode 6 (CFA4.0)
      .............
83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6
Quote from a1ex here (https://www.magiclantern.fm/forum/index.php?topic=12862.msg199697#msg199697)
Quote
@reddeercity: more details after I'll get a new card (it *is* possible to overclock the 5D2 CF interface).
You can send ATA commands to the CF card (QEMU emulates them), so if the only difference between UDMA 6 and 7 is timing, it might even be possible to put the card in UDMA7.
If this could work , then we would have some write speed as the 5d3 or close to it (120MB/s) anything we could gain is better then 79MB/s
some related stuff --> https://www.magiclantern.fm/forum/index.php?topic=21311.msg204395#msg204395

Lots to do yet ,  but we are on the way  :D
Edit: just notice the crop factor when down from almost 3 (2.8 ) to 1.95 ,
which mean it's more usable  , e.g. 24mm -> 46.8 mm
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 03, 2018, 09:07:35 AM
@reddeercity
thanks for this detailed feedback!

Quote
if you can fix the compiling problem/error in mlv_lite
yes , I'm working on it.

Quote
Now next we need compression , that's lossless compressed raw
yes, I saw your work with A1ex but but it's not my speciality yet.

Quote
increase the CF card write speed
yes , good idea . but compressed raw wouldn't be good enough  ?

Quote
crop factor when down from almost 3 (2.8 ) to 1.95 , which mean it's more usable
yes, so, don't you think that it remains also and above a problem with frame height?

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 04, 2018, 12:54:14 AM
@reddeercity
thanks for this detailed feedback!
yes , good idea . but compressed raw wouldn't be good enough  ?

No we need the higher write speed , compressed raw would give us (at .5 compression , which I read would the average)
from http://rawcalculator.netlify.com/calculator_desktop
basic 10bit just out of reach of cf/5d2-d4 cams
Code: [Select]
Resolution:2880x1080 10bit
 Data rate:88.9MiB/s
 Frame size:3.7MiB
 Crop factor:1.96x crop
 Field of view:20.8°
 RECORD TIME: 5 minutes 43 seconds per 32GB card.
14bit compressed raw at .5 compression
Code: [Select]
Data rate:62.2MiB/s
So this would be the max if lossless worked on 5d2 , better crop factor
Code: [Select]
Resolution:3200x1080
Data rate:69.1MiB/s
Frame size:2.9MiB
Crop factor:1.76x crop
Field of view:23.1°
 

16x9 2880 with increased height & compressed raw , still out reach of the current CF D4 bandwidth (75-80MB/s)
need faster interface --  overclock the cf bus driver or find away to run cards at udm7 , theoretically @ umd7 the write speed should be 120-133MB/s
Code: [Select]
Resolution:2880x1556
Data rate:89.7MiB/s
Frame size:3.7MiB
Crop factor:1.96x crop
Field of view:20.8°
 
Max rez with compressed raw with increased height at the current cf bandwidth (75MB/s)
Code: [Select]
Resolution:2880x1200
Data rate:69.1MiB/s
Frame size:2.9MiB
Crop factor:1.96x crop
Field of view:20.8°

I prefer for more horizontal rez , thou there nothing wrong with increasing the height to the max .

This is what I'm hoping to reach as a goal for raw video , it could happen only with 2 things
compressed raw (.5) & faster UMD 6 ->7 & or overclock the cf interface on the write side
Code: [Select]
Resolution:3840x1556
Data rate:119.5MiB/s
Frame size:5MiB
Crop factor:1.47x crop
Field of view:27.5°

Now , I basic all this on continuous recording for raw video  @ 24p
If some user what to use the full width at lower frame rate with increase height at maybe 10 of 12 fps
for hyperlays etc... will that's great , so there's really should be no limit on resolutions just
limitations on the presets in the crop_rec.mo .

At least now I can say there's no more aliasing & moiré patterns  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on September 04, 2018, 08:18:42 AM
yes , good idea . but compressed raw wouldn't be good enough  ?
Idea. What are the analog gain registers for 5D mark II? Dialing them down will darken output but also reduce bitrate in a lossless manner. For the 100D:
Code: [Select]
switch (crop_preset)
{
case CROP_PRESET_2K_100D:
case CROP_PRESET_3K_100D:
case CROP_PRESET_4K_100D:
{
/* assuming FPS timer B was overridden before this */
                int fps_timer_b = (shamem_read(0xC0F06014) & 0xFFFF) + 1;
                int readout_end = shamem_read(0xC0F06804) >> 16;    /* fixme: D5 only */

                /* PowerSaveTiming registers */
                /* after readout is finished, we can turn off the sensor until the next frame */
                /* we could also set these to 0; it will work, but the sensor will run a bit hotter */
                /* to be tested to find out exactly how much */

/* will add 10bit through analog gain */
adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x46};
adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x47};
adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x46};
adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x45};

/* even lower bits */
/* adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x14};
adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x15};
adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x14};
adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x13}; */

                adtg_new[4]  = (struct adtg_new) {6, 0x8172, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (6D/700D) */
                adtg_new[5]  = (struct adtg_new) {6, 0x8178, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3/6D/700D) */
                adtg_new[6]  = (struct adtg_new) {6, 0x8196, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3) */

                adtg_new[7]  = (struct adtg_new) {6, 0x8173, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (6D/700D) */
                adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3/6D/700D) */
                adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3) */

                adtg_new[10] = (struct adtg_new) {6, 0x82B6, nrzi_encode(readout_end - 1) }; /* PowerSaveTiming ON? (700D); 2 units below the "ON" timing from above */

                /* ReadOutTiming registers */
                /* these shouldn't be 0, as they affect the image */
                adtg_new[11] = (struct adtg_new) {6, 0x82F8, nrzi_encode(readout_end + 1) }; /* ReadOutTiming */
                adtg_new[12] = (struct adtg_new) {6, 0x82F9, nrzi_encode(fps_timer_b - 1) }; /* ReadOutTiming end? */
}
                break;
}
    }

This part creates lossless:
Code: [Select]
/* will add 10bit through analog gain */
adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x46};
adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x47};
adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x46};
adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x45};

/* even lower bits */
/* adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x14};
adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x15};
adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x14};
adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x13}; */

Checking your code maybe put in the analog gain registers here?:
Code: [Select]
    /* these should work on all presets, on all DIGIC 5 models and also on recent DIGIC 4 */
    if (1)                  //waza57 here, I don't think this have an effect on 5D2 until we find proper regs
    {
        /* assuming FPS timer B was overridden before this */
        int fps_timer_b = (shamem_read(0xC0F06014) & 0xFFFF) + 1;
        int readout_end = shamem_read(is_digic4 ? 0xC0F06088 : 0xC0F06804) >> 16;

        /* PowerSaveTiming registers */
        /* after readout is finished, we can turn off the sensor until the next frame */
        /* we could also set these to 0; it will work, but the sensor will run a bit hotter */
        /* to be tested to find out exactly how much */
        adtg_new[4]  = (struct adtg_new) {6, 0x8172, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (6D/700D) */
        adtg_new[5]  = (struct adtg_new) {6, 0x8178, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3/6D/700D) */
        adtg_new[6]  = (struct adtg_new) {6, 0x8196, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3) */

        adtg_new[7]  = (struct adtg_new) {6, 0x8173, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (6D/700D) */
        adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3/6D/700D) */
        adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3) */

        adtg_new[10] = (struct adtg_new) {6, 0x82B6, nrzi_encode(readout_end - 1) }; /* PowerSaveTiming ON? (700D); 2 units below the "ON" timing from above */

        /* ReadOutTiming registers */
        /* these shouldn't be 0, as they affect the image */
        adtg_new[11] = (struct adtg_new) {6, 0x82F8, nrzi_encode(readout_end + 1) }; /* ReadOutTiming */
        adtg_new[12] = (struct adtg_new) {6, 0x82F9, nrzi_encode(fps_timer_b - 1) }; /* ReadOutTiming end? */
    }

White level will have to follow but this can be altered on the fly in Mlv App, or else some if statement in raw.c or maybe building a crop_rec.c header file...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: yobarry on September 05, 2018, 07:54:01 AM
Looking up now ,  :)
I'm taken baby steps , able to increase the height now with
c0f06088 , A & B timers , target height is 2160 -- I'm at 1760


I'm a big fan of the work you've been doing for the 5D2! When I check back into the ML forum it's basically just to see what progress you're making. Big thank you for the recent achievements! How's the working coming along with increasing the height? Have you been able to save any footage at 1760px?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 05, 2018, 08:07:41 AM
Don't take this the wrong way but ,
Like I said before Lossless is not working period so the above is useless at this point , only uncompressed 10,12 & 14 bit works
For this thread 10bit uncompressed is what discussed as the norm . 

On the crop_rec.mo , I notice you can't change the ISO (at least didn't re-fresh in liveview) had to go to 10x adjust ISO then back again
before it would take effect in the crop_rec.mo 3.5k preset -- also tried to connect my hdmi zacuto evf while the in 3.5k preset , the screen what back
on the evf and recorded a few seconds and it change rez to 1856x1006 by it self . Un plugged the hdmi the lcd screen was back also .
flip back & fouth in 3x to 1:1 thern back to 3x toggle the crop_rec.mo off and on a few  then is was back to normal .

I think I know my problem with raw.c & "raw_slurp" I'm still using 0x02 as the write channel like the 12bit I use in 1:1 and I see in the Edmac.mo
I need to be on 0x0 .
I want to reproduce the experiment's from a1ex , like full width (5632x1074) & a1ex said he had figured out a 720p 50fps mode (1856x704 at 50.004 FPS)
Yes, I see it after getting out of LiveView and back. Works as expected.

Looking for a 720p mode on 5D2? Now we need to figure out how to reduce the resolution in order to increase the frame rate :)
    ...................
edit: reducing resolution seems to work, to some extent, LiveView is frozen, camera didn't crash after ~
1 minute (didn't try more), there are some black level issues, mlv_rec shows 1856x704 at 50.004 FPS.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 05, 2018, 08:12:58 AM
I'm a big fan of the work you've been doing for the 5D2! When I check back into the ML forum it's basically just to see what progress you're making.
Big thank you for the recent achievements! How's the working coming along with increasing the height? Have you been able to save any footage at 1760px?
No not yet , test out the 2880x1080 24p preset from waza57 the least few days , I posted video links to prores files & test results here (https://www.magiclantern.fm/forum/index.php?topic=11205.msg205674#msg205674)
I'll be back at it in few days  :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 05, 2018, 08:14:26 AM
@Danne
Thanks , I do not know this . I'll try to understand.
This is it is noted in TODO list (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205624#msg205624)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 05, 2018, 08:36:45 AM
@Reddeercity

Ok, so,  if I understand ( my English is difficult) correctly:

with crop_rec , you can't change ISO settings and you have a black screen when you connect HDMi object?

question:
Is this the same when we use a 5D3 (with his crop_rec enabled)?

I need to know this for investigation , thanks.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 06, 2018, 08:08:37 AM
@Reddeercity
with crop_rec , you can't change ISO settings and you have a black screen when you connect HDMi object?
Yes that right , ISO can only be adjusted outside of 3x crop_rec like 10x zoom and is applied correctly .
There seem to be no HDMI buffer , if you do a image_dump in the debug menu the HDMI dump is black  ,
there data in the dump image but all black . Once I connected the HDMI the Overlays where still there
I could go in to ML menu's etc. ..... when I disconnected the HDMI the LCD back screen was black .

If I remember right I disabled the crop_rec to get back the screen , once that happen things where back to normal
and could use crop_rec again without any problems .

Just looking at the Edmac.mo , your read buffer
Code: [Select]
4096x4096which I thing is 32MB , that good
write channel "0" connection#5
with a buffer of
Code: [Select]
7200x1896
Now playing around with HDMI again -- will , sometime the hdmi image is pink noise and another had the frozen liveview
I think maybe there too much over head with HDMI , I did record a 170 frames with HDMI connected , the mlv recording was all messed up
I'm rendering a h264 version out M05-2314.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M05-2314.mov) (15Mb)
Liveview dump with HDMI
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fbitbucket.org%2Freddeercity%2Fmagic-lantern_10-12bit%2Fdownloads%2FLV-000_small.png&hash=91b916153d4eb63a5cc97e3418e2b729)
This is what the HDMI was showing , I took a photo after I dump the liveview image
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FeFNhje%2Fhdmi_small.png&hash=e48744159390af0d773a392681dac327) (https://imgbb.com/)
here the LV-000.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/LV-000.422) with HDMI connected & the HD-000.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/HD-000.422) dumps

Another weird thing , if you disable the overlay to "Always" the Liveview freezes , but still able to record raw video
When you re-enable the Overlays the Liveview is back to normal .
got a idea , switch GD to allow is what happen , I thinking liveview will be not frozen no didn't work
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 07, 2018, 12:46:11 AM
OK, i added all this in my list of more and more things to do.
My version of crop_rec is essentially based on things known for a long time (see here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205624#msg205624) for reminder)
I also think that A1ex seemed to be about to rewrite a crop-rec module that will work for 5D2 and probably more academically than I could do it.
With changes I do not know:

ADTG [100C]                                      -> I still do not see it (I guess good for the size)
C0F07150 and C0F0713C                  -> which does not change in with the code of the current crop_rec (possibly using adtg_gui.  I suppose good for the height)
"raw_lv_setedmac_patch" solution? -> with or without CONFIG_EDMAC_SLURP? (I guess good for the memory)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 07, 2018, 06:49:39 AM
One thing I haven't tried yet was to "force hdmi to low rez."
There a option to force hdmi to 720x480 the same size as liveview , I'll try tomorrow .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 12georgiadis on September 07, 2018, 07:11:09 PM
One thing I haven't tried yet was to "force hdmi to low rez."
There a option to force hdmi to 720x480 the same size as liveview , I'll try tomorrow .
For one project, I needed a monitor HDMI + raw rec on 7D and had a black screen. Force to VGA low rez solved the problem for liveview
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 07, 2018, 08:44:48 PM
Congtartiulations waza57 and reddeercity!!! Sorry I never got involved, magic lantern development/hacking just seems to be something I can not get the hang of even a tiny bit.

THis is amazing I will try and fix my 5D mark II as soon as I can. I am truly excited.

The PNG samples on 5D2 thread have really jagged edges, could you upload some dngs?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 08, 2018, 05:10:41 AM
Congtartiulations waza57 and reddeercity!!! ....
The PNG samples on 5D2 thread have really jagged edges, could you upload some dngs?

Sure , here you go , I dumped the dng's to true 10bit .  I had to use a Old version of mlv_dump , the new one keeps upsampleing to 14bit for some reason .
the difference is size , 14bit 5.3MB & 10bit 3.7 MB
M02-1800_000000_2880x1080_10bit.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M02-1800_000000_2880x1080_10bit.dng)
M02-1810_000000_2880x1080_10bit.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M02-1810_000000_2880x1080_10bit.dng)
M02-1802_000010_2880x1080_10bit.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M02-1802_000010_2880x1080_10bit.dng)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 08, 2018, 05:20:26 AM
@waza57 , tried to "force hdmi to VGA" (720x480) just pink noise , but there is movement when recording raw video , the preview is not frozen .
Looks like frame rate (fps) is out or the size of the buffer that feed in to the encoder is messed up .
more low level investigation is needed .

Edit: checking the mlv file I recorded (2880x1080) with HDMI force to VGA (720x480)
No frame corruption as seen on the previous file I posted here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205719#msg205719) Made a quick h264 file of that recording here (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M07-2101-hdmi_forced-to-720x480.mov)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 08, 2018, 06:15:07 AM
Something interesting , I think this for D4 cams 5d2 etc. .... from http://magiclantern.wikia.com/wiki/Register_Map
Code: [Select]
0xC0F383D4 Preview area (y1 << 16) | (x1/4), similar to raw_info.active_area (5D3)
0xC0F383DC Preview area (y2 << 16) | (x2/4)
useful ? needs investigation
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 08, 2018, 06:47:27 AM
@waza57 , this could useful for HDMI preview , from my decompiled 5d2 rom
https://www.magiclantern.fm/forum/index.php?topic=21311.msg197060#msg197060
Code: [Select]
ff0316b4 FramTable == NULL
ff031b47 @DP_SetCroppingFrameLocation(X:%d Y:%d W:%d H:%d)
ff031b7c BaseW:%d BaseH:%d Angle:%d
ff031b9f @DP_SetCroppingFrameLocation FrameSize Exchange To HDMI
ff031bd8 Angle == 3600
Would be nice to know what the in the "framtable" 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 08, 2018, 08:11:28 PM
Thanks for the DNGs they look wonderful, highlights ont he first one are smooth 😍
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 10, 2018, 06:06:18 AM
Notice on the 3.5k preset the image is not centered , it looks like it's Left and down of center
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 11, 2018, 06:30:15 PM
Can you not move it like in normal crop recording?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 13, 2018, 03:31:56 AM
Can you not move it like in normal crop recording?
No , I think because the crop windows has increased from default .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 14, 2018, 05:42:44 AM
I'm taking a break from magic lantern , so don't expect any update .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on September 14, 2018, 08:03:27 AM
I am really sorry to hear that, Reddeercity.  Now, that you achieved such a remarkable progress with the 5D2, provided so much useful information for all D4 cameras and there is finally hope for 4K-crop recording on them.  I understand, you feel humiliated but please be aware that your efforts are greatly appreciated by many people on the forum.  That is why, I hope, you will reconsider your decision and will continue providing updates on your work. 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 12georgiadis on September 14, 2018, 09:55:21 AM
@Reddeercity, you helped us a lot with your research. Please don't give up. You've done an amazing work for D4 improvement.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on September 14, 2018, 08:19:20 PM
@reddeercity: sad to read that. I enjoyed every day reading your messages, what you found out. You are so close now - the goal should not be far! :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on September 15, 2018, 12:24:31 AM
@reddeercity Thank you for all you have done! I can imagine you are tired, even from the occasional checking I could see it was a lot you were doing. Really excited to get back to this stuff when I get my camera back :D :D :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 16, 2018, 07:05:41 AM
with the value of  1000102,  I have get a big improvement of around 30% speed write.
Sadly , this corrupt the card filesystem .  ( i can't read mlv file in card and i must format the card if i try to delete it)
........hey , reddeercity, do you hear this?  ;)

Yes , nice -- Can you do a benchmark in cam with out it corrupting the file system ?
would be nice to see a LOG file . either in dm-spy or mmio to see what the card connects at .
I'm wondering if there more to UDMA7 then the reg. e.g. CF Timing
Code: [Select]
83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6So I would expect to see
Code: [Select]
DatMod = 7assuming this is UMDA but not too sure what to see for
Code: [Select]
DatTim = 3maybe ?
Code: [Select]
DatTim = 4A LOG file would confirm this
Or does the cam go to auto mode went UDMA7 is forced
This is from dm-spy log reviewing a cr2 or a h264 .mov file , not sure which one .
Code: [Select]
825FE>  CSMgrTask:00095f98:00:00: *** register_interrupt("CFDriver", 0x82, 0xffb8b8cc, 0x0), from ffb8bb58
834A0>  CSMgrTask:ffb8a92c:22:05: [ID:Model Number] = LEXAR ATA FLASH CARD                   
834C8>  CSMgrTask:ffb8aabc:22:05: IDE = 4, PCMCIA = 80, UDMA = 6
83572>  CSMgrTask:ffbdb8a0:22:01: cfDecideTiming: UDMA Mode 6 (CFA4.0)
83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6


To all the core supporters  , THANKS
I'll keep checking in .
Later I'll post a explanation for my decision to pull back from ML 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 16, 2018, 09:01:07 AM
The hardware only cares about what gets written to MMIO registers (or DMA channels). What hardware we've got here?
- CF controller (from camera); likely that's where the data transfer clocks are generated
- CF card itself (this one may also have to know about UDMA mode; it's configured with IDE commands, also sent via MMIO)

What gets into debug messages (DatTim, DatMod, UDMA Mode) are just some variables in Canon code that only live in the ARM processor, i.e. they don't reach the hardware. Sure, their value decides what reaches the hardware, but in many cases we can overwrite that later (and that will happen without changing the original debug messages). The hardware doesn't look at debug messages - these are just human-readable messages written by Canon engineers for themselves.

So, you won't necessarily see DatTim = 4 or DatMod = 7, unless you patch Canon code to print that. Either you change that message (easy, but the hardware won't listen), or you change the message AND add the code to handle these values (not present in the original Canon firmware), or - sometimes easier - you just write a different value to these registers later. That last approach (i.e. changes done via DIGIC Poke) won't show up in a regular DebugMsg log, but will show up in a MMIO log.

So, the first half of the story is probably covered by the registers waza tried. We know they are somehow related to clock generation, lower numbers appear to result in higher frequencies, but that's pretty much it.

The second half of the story may or may be necessary. Some cards are able to work at (slightly) different speeds than what was declared in the IDE commands, others maybe not.

Sorry I'm not always able to give a complete answer quickly enough. Some topics, such as this one, require experimenting, as I don't know the answer in advance. I'm physically unable to do a fair "round robin" over all the active ML topics during let's say one week, as that would require too many "context switches" for me (and I'm not good at that), so inevitably some topics are going to be delayed. Seeing a message (with new useful content) on some topic reminds me to look into it; seeing the same message in 3 different places... triggers a priority drop on my side. Others may react in different ways; no need to get mad at them just for that.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on September 16, 2018, 09:31:47 AM
@a1ex +1

There is nothing personal about getting a little forum reminder now and then. I like the fact that Audionut is checking in on things. Keep up the good work everyone.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 17, 2018, 05:17:29 AM
I did a head shot to see how bad with frame skipping enable is , usable but the frame drops are noticeable
had to cut up the audio a lot to make it sync up , it's still not 100% but close (be kind please)

I what to show the concept of near UHD (3k) 10bit raw video (thanks waza57  :D)

2880x1080 23.976p 10bit , EF 24-70mm @ 24x 1.95 crop factor = 48.6mm , 1/42th sec shutter , 400 ISO
mlv_dump to true 10bit  exported from Resolve 12.5 to rec709 DNxHR 444 , finished off in FCPX uploaded file ProRes422Hq
Hopefully this will encourage other 5d2/d4 cam user to join in

Since Youtube doesn't have 2k or 3k options it only play at 1080p
so I upload a mp4 to my goggle drive at the native resolution (2880x1080p) 600MB
https://drive.google.com/file/d/1obTz9AyEglhv9u3wjKS6x2mDWVxdkU79/view?usp=sharing

@a1ex thanks , but I feel I need to clear up something here first before I continue on (It's too important now to drop this development)
I think there's being a big miss understanding here ,
My original intent was not spam you with new threads & questions , My logic with to put all the CF card research for d4 cams in one place without
trying find little bit of info here & there and the UHS-I / SD cards investigation thread , to me that was the wrong thread to continue on
the discussion being it not cf card . So I could not understand why that first thread was locked , to me it was a road block for some reason towards me .
specially after I thought I made my self clear in the second thread I start with the same topic heading , and that turn in to a shit show as you know .

I still think a separate thread for CF card development is needed , but that my own opinion .
I'll just work from this thread for now on .
Sorry for the confusion .


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Audionut on September 17, 2018, 07:16:42 AM
specially after I thought I made my self clear in the second thread I start with the same topic heading , and that turn in to a shit show as you know .

I believe that I need to set a standard, publicly.

http://www.catb.org/esr/faqs/smart-questions.html#keepcool
Quote
Much of what looks like rudeness in hacker circles is not intended to give offense. Rather, it's the product of the direct, cut-through-the-bullshit communications style that is natural to people who are more concerned about solving problems than making others feel warm and fuzzy................it may help you cope with our eccentricities if you think of us as being brain-damaged....

I attempted (I am not a robot, I am fallible) in the first thread to show some friendliness.
Sorry my reply wasn't useful.

Regards,
Audionut.

The second thread, was pure emotion.  It was made public, and it was dealt with publicly (see first line of reply in this post).  Taken to PM would have been significantly better.

I have critically maintained this public standard with you in the past, but having said that, at any point in time..........I could have also handled the matter in a more private nature, showing more respect to your contributions to the community.

Emotions take time to process from this keyboard, so I (unfortunately on some occasions) revert simply to,
Quote
Exaggeratedly “friendly” (in that fashion) or useful: Pick one.

I'm sure somehow, at some stage in the future, that I can be better at being useful, and also a little (more) friendly at the same time.  Maybe we could start some banter in general chat or something, where we can all just shoot some shit (https://www.urbandictionary.com/define.php?term=shoot%20the%20shit).  We could discuss topics in this thread, such as, hey Audionut, I'm trying to start a new thread regarding CF cards, not SD cards, you dumb shit!  Why you close my thread for?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on September 17, 2018, 06:28:54 PM
I still think a separate thread for CF card development is needed , but that my own opinion .

A while back my opinion was that in order to get crop_rec_4k working on Digic 4 cameras we needed to figure out 10bit/12bit (https://www.magiclantern.fm/forum/index.php?topic=5601.0), lossless compression (https://www.magiclantern.fm/forum/index.php?topic=18443.0), get the basic crop_rec module (https://www.magiclantern.fm/forum/index.php?topic=17021.0) working then finally tackle crop_rec on steroids (https://www.magiclantern.fm/forum/index.php?topic=19300.0) and speeding up the SD/CD read/write speeds (https://www.magiclantern.fm/forum/index.php?topic=12862.0). Instead of trying to follow so many development discussions that had only a few posts related to Digic 4 cameras, I thought it might be better to keep all Digic 4 advanced development on this topic but I got chased off (https://www.magiclantern.fm/forum/index.php?topic=19336.msg194212#msg194212).

Hey, you did a great job with the theory and @waza57 figured out the code and posted a test build for 5D2 users (https://bitbucket.org/waza57/magic-lantern/downloads/). There are several other cameras that could benefit from your work and there is still a lot more to do before Digic 4 can be merged into the crop_rec_4k branch so let's not hold a grudge and keep the momentum going.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 23, 2018, 07:54:33 AM
Find a little more info on CF cards , https://www.compactflash.org/cf-cards
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FiQwRZ9%2FCF_Card_Spec_small.png&hash=e14570808ea221b5dfd9f54e83a6e016) (https://imgbb.com/)

Code: [Select]
83572>  CSMgrTask:ffbdb8a0:22:01: cfDecideTiming: UDMA Mode 6 (CFA4.0)So there the limit on the 5D2 , the card connects at udma6 @ CF4.0 = about 90MB/s

Does the 5D3 @ udma7 run at CF6.0 ?
Should be able to tell from a dm-spy log , I found mind after reviewing a cr2 file .
Can any 5d3 user confirm this please .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on September 23, 2018, 08:01:00 AM
Does the 5D3 @ udma7 run at CF6.0 ?

I bet it does 6.0 or 6.1:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi781.photobucket.com%2Falbums%2Fyy98%2Fblitzkrieg669%2F5D%2FBENCH_SDCFXPS-064G_zpsf97r5i3e.png&hash=dae2bb01106ab4c6d8849cf43045b003)
Screenshot by blitzkrieg666 in message https://www.magiclantern.fm/forum/index.php?topic=12630.msg161435#msg161435
Close numbers to firmware 1.1.3 as seen in https://www.magiclantern.fm/forum/index.php?topic=12630.msg144600#msg144600
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 23, 2018, 08:41:49 AM
Thanks  Walter , yes more then likely .
Found a very complete CF Card specifications pdf from 2005 , a little old (CF+ & CF SPECIFICATION REV. 3.0)
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/cfspc3_0.pdf
On page 67 there is Timing Modes definitions
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgWFH7U%2FCF_Card_Spec_timing_mode_small.png&hash=8755cd8629aa5894f0e1c0a7020761b1) (https://imgbb.com/)
Code: [Select]
83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6From this it looks like "DatTim3" = 100 ns  , So does that mean 5d3 run in Mode 4 @ 80ns ?
I maybe wrong here , but it sure looks like it -- did waza57 test with udma7 on 5d2 need to adjust the mode time to shorter  e.g. 80ns ?
I my need to run some test in forced udma7 and see if I can get some logs

Edit:On Page 88-- noticed that CF cards can be run @ 8bit data & or 16bit data from the pdf , so maybe the d4 cams are in 8bit data write mode instead of 16bit data write mode
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on September 23, 2018, 08:54:48 AM
IMO (see page 2) all specs by CFA are copyrighted (outdated or not) and links to it might be able to harm ML site owner.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 23, 2018, 09:17:21 AM
Ok , I put the PDF file on my bitbucket downloads & if that's not ok then I'll remove the link
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 25, 2018, 01:00:29 PM
Some notes on fixing the live preview on 60D, after resolution overrides:

x5, x10:

C0F08518: 0x045109D7
C0F0851C: 0x00D601E6
C0F08520: 0x038A05FE

To find them: register diff with adtg_gui between two x5 configurations shifted horizontally by one position
(careful not to shift the captured image position; only the preview must be shifted).

Interpretation:
0x38A05FE - 0x00D601E6 => 1048x692.
"HD" YUV422 buffer size in these modes: 1024x680.

=> C0F0851C/C0F08520: output resolution, with some margins; these might be trimmed later.

Changing the lower half of C0F0851C and C0F08520 by the same amount: horizontal translation.
Same for the upper half: vertical translation.

C0F08518: 0x045109D7 = 2519x1105.
From C0F06084/88: 2520 x 1105. So, register C0F08518 specifies the "input" resolution (i.e. captured Bayer data). Subtract 1 on the X axis.

Increase X resolution by 1 unit from C0F06088: 0x45205B6 -> 0x45205B7 (1 unit = 2 pixels)

Fix the preview by adding 2 units to C0F08518: 0x45109D7 -> 0x45109D9.

Register C0F08518 is not used in 1x. It also appears in 640x480 crop mode, with value 0x026F0397. Raw resolution in this mode is 920 x 623. Check.

Register C0F08518 also appears on 5D2, 5x, with value 0x04670907. Raw resolution 2312 x 1127. Check. To fix the preview, C0F08188 has to be updated as well.

It also appears on 700D, CR2 capture, value 0x0DC80A4F. Raw resolution 5280 x 3529. Check, after adding 1 and multiplying by 2 on the X axis.

It does not appear on 700D in LiveView. It does not appear at all on 5D3.

Related: Bilal notes for 700D / mine for 60D: https://www.magiclantern.fm/forum/index.php?topic=9741.msg203674#msg203674

TLDR: looks like we've got a way to:
- specify the RAW (Bayer) input buffer size (for the image processing path that creates the live preview image in x5)
- pan the preview window around (without resizing it)

For image resizing: use adtg_gui to "diff" the registers between x5 and x10, then lock their values. Preview in x5 and x10 will stay the same. Figure out the meaning of these registers to find out how to apply arbitrary scaling to the preview image.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 25, 2018, 10:18:28 PM
Please, What are the problems fixed here?

1 - The size of the preview (before or during the recording)?
2 - The preview is frozen or very slow ((before or during recording)?
3 - Bad previewing with artefacts (before or during recording)?

(On 5D2 the only issue (for me) is that the preview is too slow )

Thanks.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on September 25, 2018, 10:35:22 PM
That's right; CPU-based preview is slow, hardware-based real-time preview is... well... hard.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 26, 2018, 05:28:43 AM
In the 5d2 & 5d3.113 rom disassembly found the same "cfDecideTiming" ,
Looking for the parameter that decide timing  & UDMA mode , still no luck yet
5D2 rom disassembly
Code: [Select]
ff3db894: e28f2f55 add r2, pc, #340 ; ff3db9f0: (65446663)  *"cfDecideTiming: UDMA Mode %d (CFA4.0)"
ff3db8b8: e28f2f56 add r2, pc, #344 ; ff3dba18: (65476663)  *"cfGetRegisterTiming: I/O %dnS (CFA3.0)"
ff3db8d8: 328f2e16 addcc r2, pc, #352 ; ff3dba40: (65476663)  *"cfGetRegisterTiming: I/O 250nS"
ff3db9a4: e28f20ec add r2, pc, #236 ; ff3dba98: (65476663)  *"cfGetRegisterTiming: I/O 120nS"
ff3db9bc: e28f20f4 add r2, pc, #244 ; ff3dbab8: (65476663)  *"cfGetRegisterTiming: I/O 250nS (PIO Mode4)"

5D3 113 rom disassembly
Code: [Select]
ff727f18: e28f2f56 add r2, pc, #344 ; ff728078: (65446663)  *"cfDecideTiming: UDMA Mode %d (CFA4.0)"
ff727f3c: e28f2f57 add r2, pc, #348 ; ff7280a0: (65476663)  *"cfGetRegisterTiming: I/O %dnS (CFA3.0)"
ff727f5c: 328f2f59 addcc r2, pc, #356 ; ff7280c8: (65476663)  *"cfGetRegisterTiming: I/O 250nS"
ff72802c: e28f20ec add r2, pc, #236 ; ff728120: (65476663)  *"cfGetRegisterTiming: I/O 120nS"
ff728044: e28f20f4 add r2, pc, #244 ; ff728140: (65476663)  *"cfGetRegisterTiming: I/O 250nS (PIO Mode4)"
Interesting 5d2 nearly Identical to 5d3 113 . There's no reference to any mode higher then "cfa4.0" on 5D3.

Edit: Looks little a closer (5D2 disassembly)
Code: [Select]
ff38affc: e28f2f7d add r2, pc, #500 ; ff38b1f8: (64496663)  *"cfIdentifyDrive: Set UDMA( Mode=%d )"
 (5D3 113 disassembly)
Code: [Select]
ff6aa9dc: e28f2f77 add r2, pc, #476 ; ff6aabc0: (64496663)  *"cfIdentifyDrive: Set UDMA( Mode=%d )"
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 26, 2018, 06:06:49 AM
I may have found the "CF card Configuration base"
Code: [Select]
RequestConfiguration: pLStorage=0x884e5c
81F67>  CSMgrTask:ffb8bcc8:22:01:                       ConfigBase=0x200, StatusReg=0x0
81F90>  CSMgrTask:ffb8bce0:22:01:                       PinRepReg=0x0, CopyReg=0x0
81FB6>  CSMgrTask:ffb8bcf8:22:01:                       ConfigIndex=0x1, Present=0xf
81FE2>  CSMgrTask:ffb8bd80:22:01: RequestConfiguration: Base = 200, Data = 41
8200E>  CSMgrTask:ffb8bde0:22:01: RequestConfiguration: SUCCESS
82031>  CSMgrTask:ffb8c688:22:01: CF_RequestConfiguration: err=0x0
82058>  CSMgrTask:ffb8c6a4:22:01: ConfigIndex = 1 Match

The "ConfigBase=0x200" sure looks very close to waza57 test (https://www.magiclantern.fm/forum/index.php?topic=12862.msg206010#msg206010)
Code: [Select]
0xC062850C --> 0x1213 and I obtain UDMA 0 speed (around 15MB/s)
                     --> ...........
                     --> ...........
                     -->0x0202 and I obtain UDMA 6 speed (around 80MB/s) i suppose original value
So could 0x200 be the same as 0x0202 or close , if so them maybe 0x100 will work instead 0x1000102 ? Just a thought  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on September 26, 2018, 09:30:57 AM
I have not 5D2  ROM disassembly to compare but when i see this :

Quote
There's no reference to any mode higher then "cfa4.0" on 5D3.

I think it's a good news.

Could we think that "500" for 5D2 and "496" for 5D3 can be interpret as  parameters passed to function  "Set UDMA( Mode=%d )" ?

For 0xC062850C -> 100 , I have already tested and 5D2 freeze.



Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on September 27, 2018, 01:54:16 AM
Found a very complete CF Card specifications pdf from 2005 , a little old (CF+ & CF SPECIFICATION REV. 3.0)
Also revision 4.0 (2006) can be found just by googling, but apparently no rev.4.1a neither 6.0

Instead latest revision can be bought from CFA website (https://www.compactflash.org/cfa-specifications)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on September 27, 2018, 04:52:50 AM
@waza57 I sent you a pm , should help with post #269
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 01, 2018, 03:43:47 AM
Found some info on the CF5.0 specification , looks like it was implemented in 2010
Key Feature Benefits :
Code: [Select]
Capacity points beyond current limitation of 137GB (up to 144PB) (1,024 TB = 1 PeteByte)
& more efficient data transfer (32MB per transfer versus 128K per transfer)
Data Set Management Command/Trim (Mandatory)
More efficient cleanup of unused space on memory card (LBA’s)
Update ATA References to ATA-6 & ATA-8/ACS-2
Quote
Mr. Shigeto Kanda of Canon and the CFA chairman of the board said
“The higher capacity and higher performance of CF cards enabled by the 48-bit addressing feature in the CF5.0 specification
will further increase the value of DSLR cameras. The Video Performance Guarantee feature of the CF5.0 specification
will help CF cards to expand into new markets such as high-speed movie equipment like professional video camcorders”.

So it seems that there 48-bit addressing on CF5.0 vs CF4.0 32bit addressing
FAT File Allocation Tables https://en.wikipedia.org/wiki/File_Allocation_Table#FAT32
so can we trick 5d2 to cf5.0 for 48bit addressing , or is 48bit addressing only for exfat format ?
Still researching this .

Edit: Here the PDF file where I got the info about CF5.0
cf5_0_Spec100222.pdf (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/cf5_0_Spec100222.pdf)

found the CF6.0 Spec PDF news release
cf6_0_101118.pdf (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/cf6_0_101118.pdf)
This seems important information
Quote
CompactFlash cards are capable of 600x or 90MB/second throughput performance using the current Ultra DMA Mode 6.
CF 6.0 Ultra DMA Mode 7 along with 48-bit addressing defined in the CF 5.0 specification enables the development of
CompactFlash cards with up to 25% more throughput performance.
@waza57 This looks very close to your 30% (https://www.magiclantern.fm/forum/index.php?topic=12862.msg206061#msg206061) increase in write speed
So I guess we know why 5D2 is limited to around 79MB/s (UDMA6.0 CF4.0)

Quote
The CF 6.0 specification includes a new Sanitize Command, Trim Usage Guidelines and an Operating Temperature Range function
along with adding Ultra DMA Mode 7 which supports 167 MB/second speed.
Code: [Select]
Benefits: Ultra DMA Mode 7
Key Feature Enhancement:
Provides an interface speed of 167MB/s.
This speed enhancement enables a new generation of higher performance cards while providing complete backward compatibility.

Benefits: Sanitize Command
Key Feature Enhancement: Provides an efficient NAND Block Erase of
the entire user data area to return the CF card to “fresh” state before reuse or repurposing.
Leverages a command defined in INCITS T13 ACS-2.

Benefits: Trim Usage Guidelines
Key Feature Enhancement: Provides improved write performance consistency.

Benefits: Operating Temperature Range
Key Feature Enhancement: An optional card capability to report the
operating temperature range of the card.
This allows card/camera combinations to enable use in extreme temperatures.code]

So from this
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 02, 2018, 09:04:06 AM
Trying to reproduce a1ex's full width experiment on 5d2 , It not totally right but close
I had a problem with cmos[2]  for some reason I couldn't get 0x00e 
Quote
For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.
everything else work out , not sure why I can't , the best I could get is 0xe , So left it at default "0x10e" (will try later again on "cmos[2]" )
Did a Image dump , it did save a successful DNG & Liveview 422 image

I was able to save a mlv in 10bit 12.5fps 3520x1074 about 100 frames (max size in normal 3x crop_mode)
Here a sample frame M02-0002_00040.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M02-0002_00040.png)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fpreview.ibb.co%2FkzD2uK%2FM02_0002_00040_small.png&hash=5f7b5df7fd420ddf90891341e24ee886) (https://ibb.co/mXpaEK)

The DNG dump was 5632x1074 , 5.239 aspect ratio . RAW-004.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-004.DNG) tif file RAW-004.tif (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-004.tif)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdxthTe%2FRAW_004_small.png&hash=4db6b24a8393ac5a4632bed971170908) (https://imgbb.com/)
Active Area    :52 160 1127 5792
Code: [Select]
File Name                       : RAW-004 (2).DNG
File Size                       : 11 MB
File Modification Date/Time     : 2018:10:02 00:01:54-06:00
File Access Date/Time           : 2018:10:02 00:51:47-06:00
File Creation Date/Time         : 2018:10:02 00:51:47-06:00
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Subfile Type                    : Full-resolution Image
Image Width                     : 5792
Image Height                    : 1127
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1127
Strip Byte Counts               : 11423272
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 1277
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 5632 1075
Active Area                     : 52 160 1127 5792
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 5792x1127
Megapixels                      : 6.5

The liveview dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FbTN6Mz%2FLV_012_small.png&hash=0ce8ac8f7fc35f76182d9389c1ff16ce) (https://imgbb.com/)

I what to reproduce all of a1ex crop_rec experiment on the 5d2 , specially the 50p 1856x704 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205641#msg205641) ;D
I would like to put these resolution's in the crop_rec module at some point , even though the write speed on the 5d2 is slow right now it would be nice to have this part done.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 02, 2018, 09:27:23 AM
This is interesting , after recording that short mlv file @ 3520x1074 @12.5fps I was getting very high right speeds and I
don't change anything else
Code: [Select]
# Config file for module mlv_rec (MLV_REC.MO)
mlv.video.enabled = 1
mlv.res.x = 9
mlv.bpp = 0
mlv.aspect_ratio = 16
mlv.write_speed = 8527
mlv.preview = 2
mlv.display_rec_info = 2
So 85.27MB/s  :o , not sure why maybe 12.5 fps ? can't see it
this is a mystery why , but I like it !
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: theBilalFakhouri on October 02, 2018, 09:37:49 AM
Not really a mystery it's same in all other cameras (https://www.magiclantern.fm/forum/index.php?topic=12862.msg202507#msg202507) :P
Yes, it's normal.

If capture speed is greater than writing speed (i.e. recording is not continuous): memory bus is the main limit. If you capture at a higher data rate (e.g. higher resolution and/or FPS), there won't be enough bandwidth left for file I/O at full speed. I'm not sure how DMA priorities are working, or whether there's any way to adjust them. This is why the hacks that freeze LiveView are able to achive slightly better speed - they turn off some "useless" image streams, freeing some memory bandwidth.

Otherwise... the buffer will be mostly empty and the speed will be limited by image capture speed. In this case, mlv_lite will try to "benchmark" only when the card is actively writing, and may print the "idle" time as well. However, this method is just an approximation: there will be some overhead caused by small buffer sizes (as the raw recording task starts writing to card as soon as it has completed frames in the buffer, rather than waiting for a huge contiguous chunk to be completed). No big deal, as in this case, recording will be continuous anyway. It may affect the estimated time for the next recording.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 03, 2018, 09:59:30 AM
Yes finally 50p on the 5D2  8)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcwR66z%2FVRAM3_small.png&hash=1c140d501a247e5f0fe00a82cf6939c7) (https://imgbb.com/)

Got a1ex's experiment to run on my 5d2 , it was a little tricky , had a few freezes with c0f0713c & c0f07150
and take me a while to figure out that I need to start with 30p NTSC in the Canon Menu  :-[

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkLLute%2FVRAM2_samll.png&hash=68a637b4f4dd5ce015cb31ec7a9230ae) (https://imgbb.com/)

but after that I needed to adjust the registers in a certain order or else it would lockup/freeze
So it go like this :
Code: [Select]
CMOS[1]
c0f06088
c0f06008
ADTG12[100c]
c0f0713c
c0f07150
c0f06014
Got not too bad of a image , needs work yet to clean up totally , did a image dump , got DNG, 422 .
I was able to record a squeezed 1856x704 10bit @50p for about 1000 frames , after unsqueezed I get 1856x1172
then crop off the bottom 266 lines to get 1856x906 .Very smooth frame rate , lots of detail , can't wait to try  some slow-mo shots  :)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FeWLBrz%2F50p_1856x704_mlvproducer_file_info.png&hash=91d1e22c730389171079073ccabea1e6) (https://imgbb.com/)

Here the RAW-1880x705.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-1880x705.DNG) from the image dump ,  squeezed (1880x704)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgH3jde%2FRAW_004_3_small.png&hash=6c73cce6d45ecf7eca3ee0fdca5fd7c7) (https://imgbb.com/)

Un-squeezed to 1880x906
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjjkajK%2FRAW_004_3_1880x902_small.png&hash=4f7f46331ced31d426325c99daad0f4c) (https://imgbb.com/)

here a short 56Mb I exported from MLVProducer Un-Squeezed with 3x5 option M03-0028_1856x906_50p_h264.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M03-0028_1856x906_50p_h264.mov)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcAoT4K%2F50p_1856x704_mlvproducer_Small.png&hash=dda960179901a69b88549358115e877e) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FegwWPK%2F50p_1856x1172_mlvproducer_unsqeezed_small.png&hash=88ccbb393bb71b82551c89795ec9b06e) (https://imgbb.com/)

exported frame from mlvproducer un-squeezed & cropped
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fd6nKde%2F50p_1856x1172_after_crop1856x906_unsqeezed_small.png&hash=17bca1e3db222356b6e10dd7edf74fee) (https://imgbb.com/)

Now to get Full width working properly , that's my next challenge .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ch_d on October 03, 2018, 10:42:01 PM
OMG! ML team - You are the best!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 04, 2018, 08:22:11 AM
This is a continuation of post #278 -- Seeing how far I push the frame rate , got to 71.737fps before it froze/Locked up , had to pull battery.
I was looking for 75fps , I thinks it there just need to play around more , even thinking I can push it to 100 fps  :D 

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fb3NTpK%2F5d2_71fps_small.png&hash=d4c8bde495769ee7a0bda1aafdaf345a) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on October 04, 2018, 11:46:15 AM
Really? 70FPS on 5d mk II? Guys i'm following you here on this topic but never write. So this my first time. You are awesome.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: CITY-U1001 on October 04, 2018, 04:00:46 PM
reddeercity , show test video in slow mo :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 06, 2018, 05:33:06 AM
Will it looks like I made a mistake on the 50p experiment from a1ex & I'm happy I did !
After going thought it again , I made a error by not refreshing Liveview after setting CMOS[1]
So after the 3x5 unsqueezed  I get 1856x1172  :o  not 1856x906 .
M05-2017_1856x1172_50p_h264.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M05-2017_1856x1172_50p_h264.mov)
Got a really clean image now but black level is off a little as per a1ex post .

Here the squeezed RAW-007_1880x705_50p.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-007_1880x705_50p.DNG) from image dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhV2YuK%2FRAW_007_1880x705_50p_small.png&hash=8cd2f6b0a4a807e39a930a9aea592140) (https://imgbb.com/)

RAW-007_1880x705_50p_Unsqueezed_1880_1172_50p.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-007_1880x705_50p_Unsqueezed_1880_1172_50p.png)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FnOxcMz%2FRAW_007_1880x705_50p_Unsqueezed_1880_1172_50p_small.png&hash=516617123a85e2e67e26390a05575090) (https://imgbb.com/)

Exported frame with some CC from MLVProducer
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fdb9DuK%2FM05_2017_00036_1856x1172_50p_small.png&hash=dfd302c9b403302f77fef4b32b0ee588) (https://imgbb.com/)

Liveview 422 image dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FmeB9ZK%2FLV_016_small.png&hash=8fd665fb81dbed28022f359bbc4dfd30) (https://imgbb.com/)

Screen shot of Liveview
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhRa08e%2FVRAM9_small.png&hash=3ed25a6ed573bfb01f9ae782d927ee58) (https://imgbb.com/)

In short , This is As exciting As UHD (2880x1080) on the 5D2 or maybe better in some cases .
To sum it up we are within reach of stable 50p @ 1856x1127 (records @ 1856x705) that's better then the standard
1856x1044 @ 10 & 14bit 24fps , maybe it's time to look in to 3x1 & 1x3 binding to reduce aliasing & moire pattern .
 8)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on October 06, 2018, 12:48:35 PM
A nice step forward, Reddeercity!  Keep going!

For those of you who still question if it is worth the effort to use Magic Lantern in these old Digic 4 cameras, please watch this video:


Imagine, we also have 4K crop recording working ...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: goldenchild9to5 on October 06, 2018, 03:44:37 PM
@reddeercity That's unbelievable.. the 5D Mark 2 capable of doing slowmotion video WOW.  Great progress Team  8)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 07, 2018, 07:05:45 AM
Dialed down the frame rate to 48.015 fps have a really clean image now ,


black levels are very close "1654-16200" , seems to clean up at "1730" in mlvproducer
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fe2sN69%2F48fps_info.png&hash=b19562d68755e197ce32684d2d78a751) (https://imgbb.com/)

8 second test clip 1856x1172@48.015 h264.mov  (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M06-2218_1856x1172_48p.mov)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjgqLtp%2FM06_2218_00001_small.png&hash=37fb2bfcec2543bc940592c7a92c3966) (https://imgbb.com/)
Frame from mlvproducer with good black levels @ 1730 with some CC ,
Looks almost perfect M06-2218_000011856x1172_48p.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M06-2218_000011856x1172_48p.png)
 
So I think 48fps is a better fit for the 5D2 (black levels are better mainly), one issue off the bat I notice I was locked to 1/333 on the shutter plus had a frozen Liveview.
Thou this could be because of ADTG_GUI taken up resource's slowing down liveview I guess .
The 2 frames different is not noticeable 50->48 and I have a better black level . More work needed of course , getting closer  ;)   

Edit: here's the Cdng from mlvfs M06-2218_000000_1856x704_48p.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M06-2218_000000_1856x704_48p.dng)

Found something interesting about global shutter on CMOS sensors https://www.qimaging.com/ccdorscmos/triggering.php
Just wondering if I'm close to a global shutter with 48fps with 1/333 of a sec ?
I never thought it could be possible on a CMOS , just know that CCD could be global shutter , Thing you learn while researching  :D

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fpreview.ibb.co%2FedY1G9%2FUntitled.png&hash=51e035313c9f646b041e1677f2d82247) (https://ibb.co/j3cTw9)

from my file I'm at 3ms @ 48fps so theoretically 70-75fps is maybe a possibility or at least 60fps
Unless I'm miss understand this .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Levas on October 07, 2018, 12:57:41 PM
Interesting article about global shutter on CMOS.
But if I understand the stuff in the article right ??? they're using external light source for global shutter on CMOS.
They're using a LED or Laser which can be switched on and of fast enough for controlling exposure.
CMOS is exposed, but about only half of the shutter time is used for exposure, the external light source kicks in every frame and exposing for a shorter time then shutter time is set.
So no global shutter for normal cinema situations unfortunately, more for like fabrication/quality control and stuff.

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 08, 2018, 07:35:06 AM
Good news I got near 100fps :D Bad the image is all noise distortion  but didn't crash !

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fb3MrJp%2F99_781fps_small.png&hash=f618a532e9215644742eccdbd6b7ea7e) (https://imgbb.com/)

I still looking for stable 60fps , we do have 48p & 50p with black levels issue that dose work .

Changed Timers
Code: [Select]
c0f06008 0x23b023b -> 0x23b011d   **this alone gave me 59.94** (looks familiar , never try just that alone)
c0f06014 0x577 ->   0x348

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcDMGJp%2FVRAM14_small.png&hash=0e62ef9f10cc74f9857b24e756dae49e) (https://imgbb.com/)
Liveview is mess , I did record a few MLV's

Code: [Select]
Image Width                     : 1856
Image Height                    : 704
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Rows Per Strip                  : 704
Strip Byte Counts               : 2613248
Planar Configuration            : Chunky
Software                        : MLVFS
Exposure Time                   : 1/250
F Number                        : 4.0
ISO                             : 800
Black Level                     : 1792
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 1856 704
Calibration Illuminant 1        : D65
Active Area                     : 0 0 704 1856
Frame Rate                      : 99.781
Showing the right frame rate & black level (1792)
Ok , back to finding stable 60fps .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on October 08, 2018, 12:30:28 PM
Incredible progress!

Also what are the next steps to getting 3k centered?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on October 08, 2018, 07:59:19 PM
If you guys need help with one more 5d mark II for your tests, let me know
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 09, 2018, 07:41:24 AM
Incredible progress!
Thanks , a1ex figured it out originally now I'm hooked  on the hi-frame rate  :)
Also what are the next steps to getting 3k centered?

I think the c0f0713c & c0f07150 is not set right , from my understanding (which is very limited)
I believe this controls the window size in Liveview (I maybe wrong) but from a1ex experiment with full width
and height liveview those reg's where needed .
This should be addressed in future updates to the crop_rec.module
I what to add same different presets anyways .
Also we need to implement a1ex's new "CONFIG_EDMAC_RAW_SLURP/CONFIG_EDMAC_RAW_PATCH" into the 5d2 4k branch
that how we have 10-12bit full frame liveview builds on the experiments page.

I'm starting to come to the end of my experiments with Hi-Frames rate , I'm starting to see really clean images now (black level not as bad now)
I increased the vertical to 720p from 704 this give me after unsqueezed 1856x1200 , but it has better black levels -- there still are not right
10bit yields the best image so far, black level are 98 (should be around 112) image cleans up at black level 75
and 14bit is 1559 and cleans up at 1200 black level , before at 704 black level where all over the place and couldn't clean up .
changed
Code: [Select]
c0f06088 0x4f40432 ->0x2e40432 =>720p
Before it was 0x2d40432 => 704
FYI -- the default is 0x4f40432 =>1250x1856 , so take the first set of numbers "4f4" that's the vertical resolution
"432" the horizontal
hence 0x2e40432 =>720x1856 ,  we read this as 1856x720
I encourage other with d4 cam not just 5d2 to try out a1ex experiments , there are very simple & will help you understand
how the camera works or at least with different resolutions & frame rates
Edit: this was in 50fps , didn't try 48 yet
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 11, 2018, 05:21:34 AM
Found a module on the crop_rec 4k 5d2 branch that "cf_acc.mo , added it to the module list to complied
Did that ok now what is it ? is it for over clocking the CF card interface ?
I haven't loaded it yet just trying to fine some back ground info , looking at the code
seem to be able to bypass canon code and access the cylinder/blocks directly or something like that .
Can anybody enlighten me please  :D
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on October 11, 2018, 07:43:58 AM
https://bitbucket.org/hudson/magic-lantern/branch/hdparm

Might be useful for overclocking, too.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 11, 2018, 07:50:01 AM
Ok , loaded it up and it reads the CF Card info
Printed this on the screen
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FbxObPU%2FTEST.png&hash=5185b4d6eaa78601fd4c2116d05b0fec) (https://imgbb.com/)

Some useful info at least , this is a Lexar 1066x 32GB
I see the card runs as a PCMCIA & the UDMA in Advance i/o --0x0003  , UDMA6 ? More then likely from the dm-spy logs
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 11, 2018, 07:51:24 AM
https://bitbucket.org/hudson/magic-lantern/branch/hdparm

Might be useful for overclocking, too.

Thanks a1ex , I'll have a look at this . :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 13, 2018, 07:08:48 AM
Might be useful for overclocking, too.
It going to take little time to get my head around "hdparm" branch for overclocking , a lot of stuff there .

Also got my new replacement CF card today (Lexar 1066x 64GB) the old one couldn't format just came up as "RAW"
So I did a fresh benchmark speed test in a few different modes to see where it at before I start to
adjust reg's to force it to UMDA 7 (I'm doing this on a new card because it I smoke it , it has warranty  :D )
1 min bench test , Movie Liveview , first run
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgbMn3p%2Fbench0-small.png&hash=80f2e8833d729b154bc2d39a39937ff5) (https://imgbb.com/)

Photo Liveview , better speed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcRgRw9%2Fbench2-small.png&hash=db0ed4cd1ffd9d3cd8b0dd42382c060e) (https://imgbb.com/)

Great Speed !! Play mode GD on
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FmdJvpU%2Fbench3-small.png&hash=8d7cafb07428aa7fc170cdb30d64ea4f) (https://imgbb.com/)

There is quite a bit of difference between those 3
So why is there so much overhead in Movie Liveview ?
Or is it just Liveview taken up that extra resources
Normally I record raw video at about 72-77MB/s on my other Lexar 1066x CF cards .
Even if we could get the write speed in play mode (87.4MB/s) for movie liveview , That's enough to record 2880x1080 continuously .

Need to do more investigation .

Edit: this was in 1080 30p fps , just checking raw video write speed --
1:1 averages 74MB/s & 3x cropmode 2144x1074 12bit  averages 75MB/s and pecks at 79MB/s
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 14, 2018, 06:35:22 AM
Working on increasing height in 3x Crop_mode  , I'm up to 2152x1331
Code: [Select]
Default Crop Size               : 2152 1331
Active Area                     : 52 160 1383 2312
Image Size                      : 2312x1383
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FfbEGg9%2FRAW-002-2152x1330-small.png&hash=f8d355ee6e7eb27d7e0e2a9e3c4d6404) (https://imgbb.com/)
RAW-002_2152x1331.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-002_2152x1331.DNG)
The last 300 line at the bottom is because I haven't implemented a1ex's redirect buffer for larger image in the edmac capture
(that's coming very soon , along with a updated crop_rec with added presets up to full width )

Also got 3200x1200 working , did have good success  with 3168x1152

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdB3j19%2FM13-2030-000000-3168x1152-small.png&hash=cea4533f54d0efb8737d7c48388a6ee0) (https://imgbb.com/)
M13-2030_000000_3168x1152.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M13-2030_000000_3168x1152.dng)

Did get 3168x1330 , still needs more work to get 24fps (at 21 fps right now)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fcnmq8p%2FM13-2141-00001-3168x1330-small.png&hash=f7b149db66a547ab361dafdb74f3bd7e) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 14, 2018, 10:02:47 AM
Try to force CF card to UDMA7 on Lexar 1066x from this post here (https://www.magiclantern.fm/forum/index.php?topic=12862.msg206010#msg206010) and here (https://www.magiclantern.fm/forum/index.php?topic=12862.msg206061#msg206061)
The Reg is only accessible with "DIGIC POKE" , there no 0xC062850C in the ADTG_GUI
Code: [Select]
0xC062850C --->1000102 As waza57 said -- corrupt the file system , the file are hidden .
I'm in the process of recovering the files ( it's not a fast process that's for sure) didn't hurt the card so far
the card is still in FAT32 , and can read -- file explorer says the card is 64GB with 7 GB in space used up with data(files)
So before I format again and install magic lantern I what to recover the setting files .
before it lost the file system I took a photo with my iPhone , and there's a increase for sure with some interesting info

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FihaLr9%2Fphoto-udma7-small.png&hash=fbaa8ad6d80b424334807f73eff84529) (https://imgbb.com/)
70.7MB/s write in the other test a few post up it was 57-63MB/s
but the read is crazy 2260.4MB/s  :o
So maybe that reg affects the "read only" ?
more investigation
Edit: Strange thing when it lost the file system it changed the name of the card to
"EOS_S_GIGIL" not sure why something with canon I guess
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: waza57 on October 14, 2018, 01:47:23 PM
@ reddeercity

Great job with fps !

Quote
"EOS_S_GIGIL"


Same thing for me...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 20, 2018, 07:57:46 AM
@ reddeercity
Great job with fps !
Thanks , a1ex did all the heavy  lifting I'm trying get to fine tune it.
There still a black level problem (green cast) , I can't fix yet -- it happens when ADTG_12[100c] 0x2 -> 0x4  is applied
(It's the one that centers the image in liveview) . If you change to 0x3 or 0x5 the image is has blue cast
So on my quest to find reg's that would help I did find the reg for black level in liveview  , but sadly it only for liveview & not for raw image in the buffer
Code: [Select]
C0F08034 0x1c00 (default)
C0F08034 0x1c00 ->1b00 (green cast)
C0F08034 0x1c00 ->1d00 (red cast)

C0F08034 0x1c00
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fg0Uk30%2FVRAM7-small-0x1c00.png&hash=0baabb1bd63b3b28d62859309c189d20) (https://imgbb.com/)
C0F08034 0x1b00
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgWaEwL%2FVRAM5-small-0x1b00.png&hash=c9d0d72d23b279a5d1e659f59dfef437) (https://imgbb.com/)
C0F08034 0x1d00
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkAMtqf%2FVRAM6-small-0x1d00.png&hash=d46be38f6adb622e05e08c2cc727add5) (https://imgbb.com/)

This what happen after ADTG_12[100c] 0x2->0x4 is applied 
but before I refresh liveview which will center the image
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fb6A030%2FVRAM11-small.png&hash=9b093ddd465911b3e69e882927058fde) (https://imgbb.com/)


I think I may have kind of figured out the "EOS_S_GIGIL"
after recovering some of the data with diskinternals.com/partition-recovery  (https://www.diskinternals.com/partition-recovery/)
the CF card came up as "EOS_DEVELOPFAT32"  , I made a disk image of the card and loaded it up in "HxD" hex editor and the beginning
of the image I have this
Code: [Select]
ëX.CanonEOS..@ ......ø..?...?...±9t.Ÿ;..........................€.)
....EOS_DEVELOPFAT32   ..BOOTDISK.FA.........................................
ëX.CanonEOS..@ ......ø..?...?...±9t.Ÿ;..........................€.)....EOS_DIGITALFAT32   ......
I'm thinking by setting
Code: [Select]
0xC062850C --->1000102 We may have accessed a canon service mode or canon development mode for maybe internal testing .
Al this only happen when I tried to record raw video , I first ran the bench.mo and it seem ok at the point.



@a1ex I'm trying to compile "hdparm" branch , getting a fatal error .
I first  tried to complied the 5d2 platform on my new VM " Up to date Ubuntu"
didn't work  , it was looking for a older tool chain "gcc-arm-nne-eabi-4-8-2013q4"
Tried to install but I don't know how to set it in the path for compiling , lucky I still have the Old Clone of
Ubuntu with all the old  tool chain pre-installed . So I complied it and all worked except  the cf_acc module fail to build
So I tried  to compile the module separately and got the following 

Code: [Select]
ml@ml-pc:~/CF_Card-Over-Clocking_magic-lantern/modules/cf_acc$ make
Updated HGVERSION
[ README   ]   module_strings.h
[ CC       ]   cf_acc.o
[ CC       ]   hdparm/identify.o
hdparm//identify.c:7:25: fatal error: linux/types.h: No such file or directory
 #include <linux/types.h>
                         ^
compilation terminated.
make: *** [hdparm/identify.o] Error 1
ml@ml-pc:~/CF_Card-Over-Clocking_magic-lantern/modules/cf_acc$

Seems to look for this "linux/types.h (https://bitbucket.org/hudson/magic-lantern/src/1af20c52fbe4cc48312af61d596967a018283eeb/modules/cf_acc/hdparm/identify.c?at=hdparm&fileviewer=file-view-default#identify.c-7)" and from what I can tell there is nothing related to "Linux" anywhere
Is there something missing ? do I need to add something or should I just comment it out ?
As a test , I tried to compile on the 5D3 113 platform -- still the same error .

I notice this is on the old core , still has raw_rec.mo .
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 20, 2018, 09:08:28 AM
I knew it !
There a Service menu I found in the rom
I don't know how to access it but maybe that "EOS_DEVELOPFAT32" did something to the card .
Something along the lines like magic lantern does when it boots

Code: [Select]
"********** FACTORY ADJUSTMENT MENU VER 0.01 **********":
"0. Exit from Factory Adjustment":
"1. Leak Check":
"2. SDRAM Check":
"3. ROM Check":
"4. SDIO Implementation Check":
"5. Video Adjustment":
"6. A Adjustment":
"7. USB Check":
"8. Adjustment Data Display and Change":
"9. Check Flag Display and Initialization":
"A. ALL Check":
"D. Debug Command":
"R. Data Receive":
"Y. Input Unique":
"Z. Input_Device_Unique":
"invalid input":
"CheckPro> ":
"--- LeakCheck ---":
"--- Finish ---":
"K218 System & Display Check & Adjustment program has started.":
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on October 20, 2018, 04:50:26 PM
I shot my cat with your build. She liked 2880:1080 so much. Everything was ok but froze grayscale preview.
https://drive.google.com/file/d/1YiBTSvS1JYzhThcNcmEZbBaBjnLibsjX/view?usp=sharing
Reddeercity, you told that you finally objective here is 4096:1550.
Look, 2.64:1 - it's a very special cinematic aspect ratio.
Do you plan to bring some options to choose between 2.64:1, 2.35:1, 2:1 or maybe 16:9?
I think 2.35:1 is the most optimal option here. It could be 3642:1550 (2.35:1) for example.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 21, 2018, 05:30:51 AM
............. Everything was ok but froze grayscale preview.
right now that one of the limitation , press half shutter before recording will give a full color preview in liveview


Do you plan to bring some options to choose between 2.64:1, 2.35:1, 2:1 or maybe 16:9?
Yes , but the choices will be limited to CF card write speed . 3k to maybe 3.5k (currently at 3168x1330 @ 21 fps)
no sense trying to record 4k @ 24p for 2 seconds .
I can implement full width @ 12.5 fps (5632x1074) and full height @ 4 fps (5632x3752)
Which I plan on doing later on (in a few week or so)
I'm really focused  right now on CF card overclocking (increasing write speed beyond 80MB's)

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on October 21, 2018, 12:32:48 PM
I can implement full width @ 12.5 fps (5632x1074) and full height @ 4 fps (5632x3752)
I'm really focused  right now on CF card overclocking (increasing write speed beyond 80MB's)

Reddeercity,

If you could implement 5632x3752@4 fps that would be fantastic!  This means very smooth timelapse videos at 1 to 4 fps which really look very beautiful, especially in combination with Dual ISO. 

What about the 7D?  It writes at 90 MB/s as it is.  No overclocking would be necessary to get reasonably long recording times at the high croped resolutions.  2560x1440 at 10bits lossless and 24 fps would be a dream come through for this fantastic old camera.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 23, 2018, 09:01:32 AM
Calling all 5D2 users --  8)
magiclantern-crop_rec-4k-5D2-eXperimental.2018Oct22.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/)

Updated File
magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip)

I implemented  a1ex's redirect buffer (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257?at=raw_video_10bit_12bit_LVState)  that's used in the official 10-12bit "raw_slurp" in the  https://builds.magiclantern.fm/experiments.html for 5D2.
Work very well with 10bit 1:1 FHD 1856x1044 @ 30fps & 24fps with out issue with mlv_rec  (did a short 30 second test at both frame rates).

More important , it works great with the "Crop_Rec" module , no problem recording 2880x1080@24fps
This was the first step to bring 5D2 in to the main "crop_rec 4k branch"

There's a little interesting side benefit from using the new redirect buffer from a1ex ,
I can get 3008x1138 with the 2880x1080 crop_rec preset , just need to set aspect ratio to 1:1
go to 10x zoom then back to 5x zoom (3x crop_mode) basically refresh liveview
that resolution is a little too unstable , but it proves that the buffer is working correctly
Now I can push forward with some different preset .

So now you have stable 10-12bit in 1:1 Full HD with 3k crop_rec , more usable now .
To come full hd 48fps or maybe 50 & Full resolution 5.6k @4 fps  to start with.
Enjoy

@IDA_ML a developer with 7d really needs to implement the code ,
I don't have one so it would be nearly  impossible for me .
Sorry .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on October 23, 2018, 09:06:39 AM
Do you have your code uploaded? Would like to check it out.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 23, 2018, 09:13:02 AM
Do you have your code uploaded? Would like to check it out.
Here's the source code Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018.zip)
The changes are in
Code: [Select]
platform/5D2.212/internals.h
platform/5D2.212/stubs.S
src/raw.c

Edit: I edited the first post in thread with the latest build ,
I'll update it with information & new build when necessary
This will help make it easy to find the new builds for 3k,4k crop_rec
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on October 23, 2018, 09:29:02 AM
Thanks. By the way. Could you tell me the ISO ADTG registers for the 5D mark 2?
ADTG2[8882]
ADTG2[8884]
ADTG2[8886]
ADTG2[8888]

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on October 23, 2018, 09:24:27 PM
@reddeercity: thanks for your build! I played a bit around with it. 10 and 12 bit seems to work very stable for me. I got no issues so far, @25fps (PAL) and 30fps (NTSC), FHD and 2144x1074 working nicely. The only thing I did not get to work is 3k. What I tried:
- activate mlv_rec, mlv_snd, crop_rec
- restart camera
- start liveview
- goto 5x zoom
- select resolution 2880
- select aspect ratio 1:1
- leave ml menu
- goto 10x zoom
- goto no zoom
- goto 5x zoom
If I now record, I get 2144x1074. Or if I go into ml menu I see the same resolution.
What am I doing wrong? I tried with Canon menu PAL (25fps) and NTSC (30fps).


Edit: forget what I wrote... so stupid... never worked with crop_rec... I found it now! ;)
Edit2: oh yeah... after getting the right settings to work, the quality of the resulting clips are brilliant at 2880x1080!!! Great work!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 24, 2018, 03:49:30 AM
Thanks, as always I'm only a part or a bigger picture  ;)
Most of the credit for the crop_rec goes to waza57 for porting it to 5d2 .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 24, 2018, 06:22:12 AM
Now that I have the redirect buffer working I can test some full width in 3x crop_mode  from a1ex's experiments .
I modify the resolutions in mlv_rec , added 3008, 3840, 4096, 5632 (H)
I successfully reproduced the full width 5632x1074 @ 12.5fps , can record continuously in 10bit
Code: [Select]
For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fc1pNrV%2F5632x1074-12-5fps-M23-2045-small.png&hash=44eba7d583453680a79a54ad7cf27f79) (https://imgbb.com/)

5632x1074-12.5fps-M23-2045.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5632x1074-12.5fps-M23-2045.mov) 40Mb , Short 12 second H264 test

selected 4096 from menu to record 4k
4096x1074-12.5fps-M23-2047.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1074-12.5fps-M23-2047.mov) 30Mb , 8 second H264 test

@Danne there's no
Code: [Select]
ADTG2[8882]
ADTG2[8884]
ADTG2[8886]
ADTG2[8888]
on the 5D2 , those reg's are for Digic 5 cams only
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 24, 2018, 07:26:21 AM
Next to get full height  as per a1ex experiment , in 3x crop_mode -- 2144x1330 @ 9.98fps (hopefully I can increase frame rate)
Code: [Select]
C0F06014: 0xEDC
C0F06088: 0x0x56A04BA
CMOS[1] => F6A
C0F0713C from 0x476 -> 0x576
C0F07150 from 0x49C -> 0x59C
9.98fps

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcYVCrV%2FM23-2259-00001-small.png&hash=42653e1b5335eb29400f10b8bb6cdf36) (https://imgbb.com/)
2144x1330-9.9fps-M23-2259.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1330-9.9fps-M23-2259.mov) 40Mb H264 clip
2144x1330-M23-2259_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1330-M23-2259_00001.png) PNG frame export from mlvproducer
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 25, 2018, 07:14:14 AM
Continuing from yesterday with full height exercise ,

2144x1586
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdDv7RV%2F2144x1586-M24-2225-00001-small.png&hash=942a2d157c9719421a158be6865d0678) (https://imgbb.com/)
2144x1586-9.9fps-M24-2225.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1586-9.9fps-M24-2225.mov) 30Mb h264 clip
2144x1586-M24-2225_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1586-M24-2225_00001.png) Frame exported from mlvproducer
Code: [Select]
C0F06014: 0xEDC
C0F06088: 0x0x66A04BA
C0F0713C from 0x476 -> 0x676
C0F07150 from 0x49C -> 0x69C

2144x1842
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fg4ZFfq%2F2144x1842-M24-2227-00001-small.png&hash=f221766d1222926d7a13c011c7b30485) (https://imgbb.com/)
2144x1842-9.9fps-M24-2227.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1842-9.9fps-M24-2227.mov) 30Mb h264 clip
2144x1842-M24-2227_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1842-M24-2227_00001.png) Frame exported from mlvproducer
Code: [Select]
C0F06014: 0xEDC
C0F06088: 0x0x76A04BA
C0F0713C from 0x476 -> 0x776
C0F07150 from 0x49C -> 0x79C
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on October 25, 2018, 03:15:17 PM
@reddeercity ->  Calling all 5D2 users --  8)
Congratulations, the first step in crop_rec is here !
I tested a 10bits 2752x1080 file and it is ok but liveview is frozen during recording.
I tested a dual_iso_crop_rec mixed video but i got an "oldapi" error message on loading dualiso module. I know I'm asking a little too much for a first step ...
But it's was so existing that I tested !
Thanks
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on October 25, 2018, 03:50:10 PM
I tested a 10bits 2752x1080 file and it is ok but liveview is frozen during recording.
Japp same here. But somewhere I read "liveview is for wimps", so I thought this is state of the art... 8)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on October 25, 2018, 04:48:38 PM
Perhaps I'm a wimps ...   ??? I want to see what i'm filming  ;) ! I'm not only use my cam with a tripod  !
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on October 26, 2018, 05:24:19 PM
I must be doing something very wrong, so I'll take the hit. Please breakdown what our setting should be, I only see static lines when I try to use 3.5K 1.1 Crop setting.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 28, 2018, 07:23:05 AM
Updated the First Post with very basic screen shot tutorial
Short Tutrial for Setting up Crop_Rec Module (3.5k 1:1 Preset) on the 5D Mark ii (https://www.magiclantern.fm/forum/index.php?topic=19336.msg182476#msg182476)
Hopefully this should clear up any confusion
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on October 28, 2018, 03:53:10 PM
I don't know what I'm doing wong but I can't wait to find out, I've follow all tutorial steps. I attached photos to represent setting, also showing that in 1856x1044 1:1 and 10x zoom everything looks OK, but x5 3.5K 2880x1080 is all static lines. PLEASE HELP ME SOLVE THIS!
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FmgbJrV%2FIMG-1348.jpg&hash=05cbeab18590c8f13672d77c8f22e105) (https://ibb.co/mgbJrV) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FeNkpyA%2FIMG-1349.jpg&hash=6720a02e5c00b9f3d3ee13aa261a82fd) (https://ibb.co/eNkpyA) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FhcK4WV%2FIMG-1350.jpg&hash=5b6966b21b43153b3b1b0db08d45ee75) (https://ibb.co/hcK4WV) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FfeoRdA%2FIMG-1351.jpg&hash=70278acf04524fdb01a92844634b5580) (https://ibb.co/feoRdA) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FdyphJA%2FIMG-1352.jpg&hash=108cf9daea12d981e1b827d792665176) (https://ibb.co/dyphJA) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FbFLYQq%2FIMG-1353.jpg&hash=f52bb0e148c64c0643f40df6f1b42899) (https://ibb.co/bFLYQq) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FdvBJrV%2FIMG-1354.jpg&hash=3fbe3f9b7b980abc03a70d96a41e5e0c) (https://ibb.co/dvBJrV) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2Fnd6JrV%2FIMG-1347.jpg&hash=de6f556f504c94a85b1dd7f92ba783d8) (https://ibb.co/nd6JrV)
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 29, 2018, 04:51:22 AM
You don't follow the tutorial , I don't know how to make it any simpler .
This is bleeding edge stuff , Go back to the tutorial and match your screen shots to the
ones in the first post , Do there match exactly in the Movie Tab  ?
Hint: something to do with fps
Code: [Select]
1:For best results start with 30 fps ntsc in the canon Liveview menu
2:Load crop_rec  & mlv_rec , mlv_snd can be load but reduces the amount of frames recorded .
3:Next setup Liveview preview , This is the most important part without this set correctly it will not work !
 -Liveveiw needs to be set to ML Gray Scale-
4:Next go in to the crop_rec menu select 3.5k preset .
 -Note ! this only work when you enter 5x zoom/3x crop_mode as per the hi-lighted texts says-
Did you do this exactly ?
The frame shots tell a different story .
Only load & enable the bear minimum to run crop_rec
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on October 29, 2018, 06:23:20 AM
Wow, @Reddeercity you’re affirmative I use both FPS setting, Thanks for the smack on the back of my head. So is it necessary to use Exact FPS on 10bit 12bit? When using 5x is it recommended to use 24mm to get a 50mm framing? Thanks Again to Reddeercity, Alex, Waza57 for the phenomenal job of making my trusty 5D Mark II at least a FHD RAW recorder, with full frame sensor RAW our images will meets up to wide range of end Presentations Format.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 30, 2018, 06:52:42 AM
Working on presets , Got a nice clean 3168x1330 but sadly only 12fps at the moment (looking to get 23.976 fps)
I need to understand the A & B Timers better , this was just luck -- started at 7fps and worked my way up to 12 fps

3168x1330 @ 12fps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhOYek0%2F3168x1330-12fps-frame-M29-2249-00001-small.png&hash=17e53cf1eb4ecdc29d5597c936ad985f) (https://imgbb.com/)

3168x1330_12fps-M29-2249.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3168x1330_12fps-M29-2249.mov) h264 10 second 30Mb clip
3168x1330-12fps-frame-M29-2249_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3168x1330-12fps-frame-M29-2249_00001.png)
3168x1330-RAW-017.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3168x1330-RAW-017.DNG) from image dump
Code: [Select]
Default Crop Size               : 3176 1331
Active Area                     : 52 160 1383 3336
These are the overridden reg's ** Note** the times (6008 & 6014) are for 7fps I didn't save the timers for 12 fps

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fe3apJL%2FVRAM5-small.png&hash=5695fd96c8d25f70ae6e781178dfe573) (https://imgbb.com/)

I did find that "CMOS[1] need to set early -- if not , thing don't work right

and you get this 2.38 A.R. ( that what I'm looking for)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FgoATCf%2FVRAM6-small.png&hash=884f0c7db961e13530ce63249e7f46a8) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on October 31, 2018, 06:51:57 AM
working on a 4k preset (4096)
Code: [Select]
c0f06088 - 46a088a =>4096x1074
c0f06008 - 2770477 => 16.389 fps
Clean dng from image_dump plus clean dng's in .mlv's

4104x1075-RAW-019.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1075-RAW-019.DNG) from image_dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhLOYf0%2F4104x1075-RAW-019-small.png&hash=7a5f1d40a836eaad35dbe1b5d8c0ac18) (https://imgbb.com/)

After cropping off the  black space it's only 3718x1074 , needs more reg's added
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhsLof0%2F3718x1075-RAW-019-small.png&hash=3cb974e8485860c6e876a729b662ee27) (https://imgbb.com/)
3718x1075-RAW-019.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3718x1075-RAW-019.png)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 03, 2018, 07:09:49 AM
Ok got 4096x1074 @ 18fps working -- It's almost useable as is  :D

Needed to add CMOS[2] 0x10E ->0xE (same as full width 5632)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkCkHyL%2F4104x1074-18fps-VRAM9-small.png&hash=319ec9436c7110708e3b3219b9e23e2c) (https://imgbb.com/)

A & B Timers setting for 18 fps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhJT6Q0%2F4104x1074-18fps-setting-VRAM10-small.png&hash=0e3f749d7ed121bbbde0ed5407337ff5) (https://imgbb.com/)

For same reasons mlv_rec in crop_mode cut off the 48 pixels from 4096 to 4048 , not sure why .
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fm6xi50%2F4104x1074-18fps-M02-2240-00001-small.png&hash=d2559be874d10fb6415d43c8a408a1d3) (https://imgbb.com/)

4048x1074-18fps-M02-2240_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1074-18fps-M02-2240_00001.png)

If you look at the dng from the image _dump it's 4104x1074 and there no black space on the right side like the PNG .
4104x1075-18fps-RAW-020.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1075-18fps-RAW-020.DNG)
4104x1074_image_dump-RAW-020.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1074_image_dump-RAW-020.png) used dcraw to export a ppm them saved it to png with Irfan View

4096x1074-M02-2240_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1074-M02-2240_000000.dng) Cdng exported from MLVFS (quick mount)
there a 48pixel black area on the right , thinking the problem is mlv_rec  , I see in the code there's a 160 pixel crop , so may be change that to 100 maybe

Code: [Select]
Subfile Type                    : Full-resolution Image
Image Width                     : 4264
Image Height                    : 1127
Bits Per Sample                 : 14
       ..........
Default Crop Size               : 4104 1075
Active Area                     : 52 160 1127 4264

Code: [Select]
Black Level                     : 1001
White Level                     : 16200
This is also interesting , can the black level really be 1001 ?
Image looks great , so I guess this is ok , but strange -- I recorded  at 10bit .

Recorded a short .mlv about 10 seconds exported it to h264 from mlvproduder 4096x1074-18fps-M02-2240.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1074-18fps-M02-2240.mov)

So close now to 24 just need 6 more frames per second .

Now it time to work on Lossless compression , since I'm close to having the presets done.
So far I have ,  full liveview , 2880x1080, 3168x1330, 5632x1074 ,4096x1074
I'm thinking of added some of the near full height in 2k mode e.g. 2144x1842 or 2144x1586.
I would like to add 48 FPS in FHD (1856x720) but it's too unstable right now , need a lot of work yet.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on November 03, 2018, 11:08:45 AM
I'm waiting for yours presets with hope.
I don't know the right number you can reach with yours registers but do you think you can include standard preset 24p with 16/9 ratio and vertical resolution like 1080, 1160, 1320, 1450, 1586 ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 04, 2018, 04:40:03 AM
......  do you think you can include standard preset 24p with 16/9 ratio and vertical resolution like 1080, 1160, 1320, 1450, 1586 ?
All presets will be dependent on Card write speed and whether or not lossless compression can be implemented .
So currently write is around 77-79MB's with the odd burst at 85MB/s , the preset will reflect that .

I guess the answer is no , I will not have standard presets . But I will have one's that are close .
The main reason is that the sensor is in a non-conformed size to Video ( that why FHD is only 1880x1054 , unlike 5d3 1920x1080)
and being raw , it can be resized as much as 150% before any image break down .
There will be some at 24p , like 1080 & 1330 -- I plan on 2:1 but not sure if I will go 16/9
It takes a long time to get this sizes to work @ 24p and the more the height is increased the more difficult it becomes
But nothing is written in stone ! things can change at any time as new discoveries can happen .

After we have this 100% working then I plan on moving to Mjpeg Compression , which I hope to be able
to have Standardize Frame resolutions up to 5.6k in a .AVI format.
But don't hold your breath  :D that could take a year or more , or maybe less who knows for sure .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on November 04, 2018, 10:27:31 AM
ok, I didn't know it takes a long time to get this sizes to works.
Could you write a brief summary (a few lines) on how to find these values for 5D2 ?
I read yours threads with a1ex about the registers, but i need a summary.
Perhaps I could understand and make my own values to share with others users and increase the preset database.

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 05, 2018, 05:48:44 AM
Everything is all there in the posts , I just reproduced a1ex experiments
read this first (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205556#msg205556) then this next (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205559#msg205559) , finish off with reproducing a1ex experiments  (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205614#msg205614) and here too (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205622#msg205622) . 
Now you know what I know , Next you need to setup a environment to compile magic lantern . Depending on your OS there different ways to setup a environment
Compiling Magic Lantern on Windows 10 (using its Linux subsystem) (https://www.magiclantern.fm/forum/index.php?topic=20214.msg188142#msg188142) , Compiling Magic Lantern with Cygwin/MinGW-64  (https://www.magiclantern.fm/forum/index.php?topic=15894.msg154435#msg154435) , Compiling Magic Lantern on a Macintosh (https://www.magiclantern.fm/forum/index.php?topic=16012.0) etc. ...
I use Ubuntu in a VM (Virtual Machine) on windows 7Pro . Next Clone waza57 source code (crop_rec_4k_5D2) link is here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205624#msg205624) ,
now you need to add "raw_slup" redirect buffer to the 5d2 4k source from waza57 to get the extended image .
This committed 3e3fac1 (https://bitbucket.org/hudson/magic-lantern/commits/3e3fac1e9c8e3b2ae956a0613c1e49c705831257?at=raw_video_10bit_12bit_LVState) is the one I used , now compile . That it .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on November 05, 2018, 10:24:37 AM
Thanks for the links.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 11, 2018, 08:10:58 AM
Looking in to the HDMI resolution limitations , if everyone doesn't know on the 5D2 the HDMI output @ 1650x1080 60i 4.2.2 uncompressed (1500MB/s)
It anyone wondering why the 5d2 has this res. when standard HDMI is normally 1920x1080 or 1280x720
From what I can figure out there was no need for 16x9 format (After all 5D2 was the first to introduce Video to DSLR's) as a second thought .
So the main object was Photo's and a 3x2 format was needed for photos in  Liveview  so hence 1650x1080 .

So I connected my HDMI Evf & loaded ADTG_GUI.mo  and saved a log file
So I may have found some reg's that control this .
Code: [Select]
c0f11314: 4d70653 .... LiveViewMgr pc=ffa07c28 addr=ffca88e0 LV resolution (raw.j.height | lv.width) before upsampling?If you convert this to decimal you get 4d7=1239 & 653=1619  so 1619x1267 , look very close
Code: [Select]
c0f0713c:     4f5 ......  LiveViewMgr pc=ff8e1830 addr=8328 HEAD3 timer (ticks?)
c0f07150:     681 ..... LiveViewMgr pc=ff8e1844 addr=8370 HEAD4 timer (ticks?)
convert this to decimal  4f5=1269 & 681=1665 so 1665x1269
I could be totally wrong but that so close , now there's more then likely more reg's evolved here , it could be the hd buffer is the limitation.

There a few reason why I what to hopfully fix this issue ,
First: to get real full 1920x1080 so it would be usable without resizing .
Second: the HDMI spec 1.3a (https://en.wikipedia.org/wiki/HDMI) allowed up to 2560×1440 at 60 Hz & 1920 × 1080 at 120 Hz
Since there's 4k monitor's & recorder out there and getting cheaper everyday, well I thinks it time to look in to this to see it it's possible .

Edit: Ok I forgot to mention the hdmi 1.3a supports 8bit(24 bit/px) , 10bit(30 bit/px), 12bit(36 bit/px), 16bit(48 bit/px) up to 2560x1440 60i
Color Space: RGB, Y′CBCR 4:4:4 & Y′CBCR 4:2:2
and apparently hdmi 1.3a supports UHD 8bit(24 bit/px) 3840x2160 30p 4.2.2 & 60p @4.2.0 and also 5k 8bit(24 bit/px) 5120 × 2880 30p 4.2.2
Maybe there's hope yet for real-time hdmi UHD up to 5k preview   :))
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on November 11, 2018, 07:47:30 PM
Wow... it looks so interesting
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 12, 2018, 07:22:00 AM
I should mention that the 5D2 HDMI signal is 8bit 4.2.2 Full range (0-255) , Check it on my AJA Kona LHi video capture card .
So by this theoretically  "UHD 8bit(24 bit/px) 3840x2160 30p 4.2.2" would be in the realm of possibility .
Here's hoping   :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 15, 2018, 06:00:44 AM
Having some fun we presets ,
5632x1844 @ 4fps , got up 6.5fps then froze liveview , unresponsive so I just rebooted .
Sorry , overexposed it too much @ 800 ISO  ::)  as frame rate came down , exposure became normal

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FfRjmkL%2FM14-2138-00050-small.png&hash=13bcb71dc0d66864cb2458ab9bbafa18) (https://imgbb.com/)
5632x1844_M14-2138_00050.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5632x1844_M14-2138_00050.png)
Code: [Select]
Default Crop Size               : 5632 1845
Active Area                     : 52 160 1897 5792
5632x1845_dump_RAW-026.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5632x1845_dump_RAW-026.DNG)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fg6Xvef%2FVRAM3-small.png&hash=bdca286508f10d60069d16e9c22058bd) (https://imgbb.com/)

Modified Reg's
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjU7aC0%2FVRAM2-small.png&hash=a9dfe159e70bf6afe1233fbda27dda7b) (https://imgbb.com/)

5632x1844_4fps_M14-2138_h264.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5632x1844_4fps_M14-2138_h264.mov)
Had problem playing it in QT player , work fine in  MPV player (https://mpv.io/) -- cross platform  ;)

I sure wish I could get 24fps at this resolution , it would be prefect  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on November 15, 2018, 05:25:08 PM
Cool -) Thanks for keeping going
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 17, 2018, 07:50:18 AM
Keeping with the same height (1842) as above , but @ 4096 and see how far I can push it
So 4096x1842 I could reach 10.5fps with a little image corruption (needed to adjust 5x frame blanking to clean up)

Shown below is  4096x1842@4fps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjJYJLL%2FVRAM4-small.png&hash=bdeccbb28aacdf26625127ffc8fd9827) (https://imgbb.com/)

notice for 4096 c0f06084->30036 , where 5632 @ c0f06084->10037 -- (for anyone that's following this and what's to reproduce)
Raw resolution c0f06088-> 0x76ao88a

Changed A & B timers from image posted
Code: [Select]
10.501 fps
C0f06008 ->0x2770462
c0f06014 ->0x7f2

4104x1843_dump_RAW-028.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1843_dump_RAW-028.DNG)
Code: [Select]
Default Crop Size               : 4104 1843
Active Area                     : 52 160 1895 4264

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fkrj00L%2F4096x1842-M16-2255-000076-small.png&hash=10c400f51b2c59a6eda7d3f6613d95a3) (https://imgbb.com/)

4096x1842_ppm_dcraw-M16-2255_000076.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1842_ppm_dcraw-M16-2255_000076.png)
exported ppm file from Cdng with dcraw -- with a quick color correction in irfanview and saved as a PNG
4096x1842_mlvfs_M16-2255_000076.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1842_mlvfs_M16-2255_000076.dng)

A short 36Mb h264 clip @ 7 FPS 4096x1842_7fps-M16-2255.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1842_7fps-M16-2255.mov)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on November 17, 2018, 10:29:33 AM
Wow, i saw the h264 video.
Maybe reducing vertical resolution till 1742 (4096x1742 (2.35:1 aspect ratio) will help to increase a couple of frames per second?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 18, 2018, 08:20:37 AM
I don't thing it will help , I thinking the sensor readout may be too slow for that resolution .

Trying 3072x1200 see if I can get 24fps
Code: [Select]
CMOS[1] 0x6ea->0xc0a
c0f0713c 0x476->0x576
c0f07150 0x49c->0x59c
c0f06008 0x2770277->0x2770346  ** A-Timer **
c0f06014 0x4ff->5cd  ** B-Timer **
c0f06088 0x46a04ba->0x4e7068a  **raw resolution **


This gives me only 19.419 fps with some frame blanking corruption (not totally clean) , seem that I can't get any more .
5d2 maybe limited to only 1080 vertically , as all my efforts seem to point to this .
I keep dropping the horizontal to get as much height as possible ,
so I guess next back to 2880 and see if I can increase the height to 1200 at 24fps
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on November 18, 2018, 07:25:36 PM
reddeercis,

is it possible, in your build, to shoot 2880x1080, and shoot normal, almost 1080p in full sensor 1:1, too (in the same time, i mean using same build)?

i want to try your build, but find some test on YT, where guy sad that full sensor mode does not work when using experimental build that crops the center of frame....

maybe i did not understand it correctly..

tnx 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on November 18, 2018, 11:04:48 PM
i just try it!

2880p is so great>))

Thank you very much for such a progress!!

I dont have preview while recording. Preview is frozen and black and white, once i press record button.

Is that normal?

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 19, 2018, 04:17:24 AM
reddeercis,

is it possible, in your build, to shoot 2880x1080, and shoot normal, almost 1080p in full sensor 1:1, too
Yes , the crop rec is only active in 5x zoom mode . So you can shoot 10bit raw in 1:1 FHD also .


i just try it! 2880p is so great>))
    ........
I dont have preview while recording. Preview is frozen and black and white, once i press record button.
Is that normal?
Yes , That's one of the limitation at the moment , until liveview preview is more understood more , that's the way it is .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 19, 2018, 08:01:29 AM
Trying a more modest size to see where the limits are .

2144x1200 @ 23.976 fps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FduoHKf%2FVRAM11-small.png&hash=d7fcc375ff52ddc94ced9072a5909002) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkEszX0%2FVRAM10-small.png&hash=eb7250c35705089861e42c5b50ba7f8c) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdS53s0%2FM18-2327-000000-small.png&hash=9195fb9ca29dbcc773ce413db9efb47f) (https://imgbb.com/)
2144x1200-M18-2327_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1200-M18-2327_000000.png)

A short h264 2144x1200_24fps_M18-2327.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2144x1200_24fps_M18-2327.mov) 75Mb 25 seconds

It does record continuously @ 73.5MB/s  :D , the preview is Not broken so you can see what's being recorded
It took a while to get it work , the sensor is not centered I had to move the image to the lower half of the sensor
to clean up the image , use CMOS[1] 0xe6a -> 0xc14 , 0xc0a centers the image but has bad corruption (frame blanking issue).

I guess to sum it up ,  1080 maybe the maximum vertical resolution with extended horizontal e.g. "2880 or 3168" that can achieved .
Every effect above 1080 is meet with failure @ 3k and greater .

It Maybe that the Sensor readout is just to slow to get higher resolution @ 23.976 fps anything less is a waste of time ,
ok maybe full res 5632x3750 @4 fps .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on November 19, 2018, 10:24:00 AM
Usable 2880x1080p, with live view, so we can see what we record, is just fine. Maybe we do not need more resolution..... :P
This is still such a big progress for 5d2
2880 horizontal is really a sweet spot!

thx for responding
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on November 19, 2018, 04:12:48 PM
Great reddeercity !
Perhaps with a continuous bit rate, the same vertical 1200 pixels, the same liveview not frozen, we can find horizontal resolution for 1.85, 2 and 2.35 aspect ratio ? like 2220, 2400 and 2820. I'm afraid the last one won't be continuous. Who knows ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: poromaa on November 19, 2018, 09:40:51 PM
Impressive!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on November 20, 2018, 02:38:57 AM
Absolutely agree with Banertop, hehe..
Usable 2880x1080p, with live view, so we can see what we record, is just fine. Maybe we do not need more resolution..... :P
It's a big progress. Crop Factor is like 1.9 but it's ok for this case with 5d mk 2, i think.
Anyway, thanks for keeping going Reddercity! And of couse we are waiting for any news. When you rich 2880x1080 24fps, I try to inspire you with little videos, shot with the new build, hehe
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on November 20, 2018, 02:45:45 AM
Also we can try to fix Metabones SpeedBuster on 5dMK2 to "reduce" the crop factor.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 20, 2018, 05:51:13 AM
I might I have a fix for the frozen Liveview in crop rec 3k preset .
https://www.magiclantern.fm/forum/index.php?topic=19336.msg206539#msg206539
I haven't tried it yet , in theory it should work

Great reddeercity !
Perhaps with a continuous bit rate, the same vertical 1200 pixels, the same liveview not frozen, we can find horizontal resolution for 1.85, 2 and 2.35 aspect ratio ? like 2220, 2400 and 2820. I'm afraid the last one won't be continuous. Who knows ?
Thanks ,
That's my next step to increase horizontal resolution , I might get 2400 or maybe even 2560 with 1200 vertical @ 23.976 fps
Don't count out 2880x1080 @ 24fps to be continuous  , I being working with others on Lossless Compression ( Canon's compression for Raw Images in CR2)
Read this ProcessTwoInTwoOutLosslessPath (https://www.magiclantern.fm/forum/index.php?topic=18443.msg205333#msg205333) and here too (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198487#msg198487) , from other cameras that support this, there about 40-50% data compression
https://rawcalculator.netlify.com/calculator_desktop
2880x1200 @23.976 fps with 14bit compressed raw =>69.1MB/s -- continuous
2880x1200 @23.976 fps with 14bit uncompressed raw =>98.8/s
2880x1080 @23.976 fps with 10bit uncompressed raw =>88.9MB/s

I'm also looking in to CF Card Overclocking , seems that the bus interface can be pushed not sure how much,
hoping to get at least 120 MB/s write speed .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on November 20, 2018, 09:35:43 AM
Here's a simple task for you guys: find out how far you can push FPS timer A.

Steps:
- download raw_diag.mo and adtg_gui.mo (https://builds.magiclantern.fm/modules.html#iso-research) and load them on top of the build from first post
- enable RAW Diagnostics (Debug menu)
- enable OB zones, disable other types of analysis
- in LiveView, press shutter halfway for 1-2 seconds to run the analysis
- enable ADTG Registers, enable Advanced -> ENGIO Registers, override FPS timer A (C0F06008) and decrease its value
- notice only odd raw values (i.e. same parity as with Canon firmware) are going to give clean image on this camera
- find the lowest value of timer A that still gives clean image (pay attention to the right border in the raw_diag screenshot)
- upload the raw_diag screenshot with the optimal value (clean image) and with the first bad value (that is, optimal value minus 2)

Run this in 1080p24/25/30 (during standby) and also in x5 zoom mode.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 20, 2018, 09:45:13 AM
It's was only a matter of time before things start to came together , once I started to understand the imaging pipeline (will at least in my mind I started to understand :P)

Ok now to the detail:
2400x1200 @ 23.976 fps
With a Clean Preview No Frozen Preview !! (will mostly , still have a little frame blanking issue -- I'll it fix later)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjTiFPf%2FVRAM15-small.png&hash=b588a03cbd950deb5c00af453960aa64) (https://imgbb.com/)

@a1ex it works , no more frozen Liveview when recording raw video  -- Thank you 8)
It took a while to find c0f08518 , it was deep towards the end of all the reg's
Wasn't should if I needed to add c0f08184 , but I did
Code: [Select]
c0f08518 0x4670907 ---------->0x4e20a07
c0f08188 0x907 -------------->0xa07
c0f08184 0x467 -------------->0x4e2
Code: [Select]
0xa07=>2567 -- horizontal
0x4e2=>1250 -- vertical




(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Ff3CCn0%2FUn-Frozen-preview-IMG-2671-small.png&hash=6bbae8be843f81d2a2295c7961ffd98c) (https://imgbb.com/)
iPhone video showing unfrozen Liveview when recording raw video -- 2.4kx1.2k-Un-Frozen-preview-IMG_2671.MOV (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2.4kx1.2k-Un-Frozen-preview-IMG_2671.MOV)

Code: [Select]
Raw Resolution 1253*2568 (2568x1253)
c0f06088 0x46a04ba ---> 0x4e8053a
Code: [Select]
Default Crop Size               : 2408 1201
Active Area                     : 52 160 1253 2568

Here is a screen shot from the frame over ride menu showing timing info
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkPBKjf%2FVRAM14-small.png&hash=8dedb371677b05c899ab96924a85691e) (https://imgbb.com/)

Proof , but still need to fix black levels -- h264 5 seconds clip 2400x1200-24fps-bad-black-levels-M20-0131.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2400x1200-24fps-bad-black-levels-M20-0131.mov)

2400x1200-bad black levels M20-0131_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M20-0131_00001.png)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 20, 2018, 09:48:27 AM
@ a1ex I posting when you posted this , I try this tomorrow -- should have time then  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 21, 2018, 06:40:58 AM
Here's a simple task for you guys: find out how far you can push FPS timer A.

Steps:
- download raw_diag.mo and adtg_gui.mo (https://builds.magiclantern.fm/modules.html#iso-research) and load them on top of the build from first post
- enable RAW Diagnostics (Debug menu)
- enable OB zones, disable other types of analysis
- in LiveView, press shutter halfway for 1-2 seconds to run the analysis
- enable ADTG Registers, enable Advanced -> ENGIO Registers, override FPS timer A (C0F06008) and decrease its value
- notice only odd raw values (i.e. same parity as with Canon firmware) are going to give clean image on this camera
- find the lowest value of timer A that still gives clean image (pay attention to the right border in the raw_diag screenshot)
- upload the raw_diag screenshot with the optimal value (clean image) and with the first bad value (that is, optimal value minus 2)

Run this in 1080p24/25/30 (during standby) and also in x5 zoom mode.

Mission accomplished
ob-zones.rar (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/ob-zones.rar)

 default                                                                                                                     
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fc4VJCA%2Fsmall-ob-zones-30p.png&hash=3dbd122fd37ae603461712cf62c5456c)1080 30p (https://imgbb.com/)  (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FjY7Rzq%2Fsmall-ob-zon-1-24p.png&hash=44f9e82114d501467f5e745d792c362d)1080 24p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FhJRx5V%2Fsmall-ob-zon-3-25p.png&hash=ef03888f53b68e2a8c2f2e7c6f1b4706)1080 25p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fee4Oeq%2Fsmall-ob-zon-2-640x480-30p.png&hash=6ee0e32406f66f340f8e4a7f9b039a75)480 30p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdCc9Kq%2Fsmall-ob-zon-4-3xcrop-30p.png&hash=fbcda96f655ac0ac1ff14b907c4bb25d)3xcrop-30p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FmNqeKq%2Fsmall-ob-zon-5-3x-Crop-24p-over-ride.png&hash=132d3b750a2470d9a23834838953b4a1)3xcrop-24p (https://imgbb.com/)
------------------------------------------------------------------------------------------------------------------------------------------------------

Modified Timer A c0f06008
NTSC 30p
1st: 23b0219=>31.864 fps -- good    2nd: 23b0217=>31.982 fps -- bad
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkQRN5V%2Fsmall-ob-zon-6-30p-31-864fps-timer-A-23b0219-good.png&hash=9f05d37cdae993ca2b2e46e05b45fa44)1080-30p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FiEWpkV%2Fsmall-ob-zon-6-30p-31-982fps-timer-A-23b0217-bad.png&hash=8fed89678fdfec2f15ab0db1d6e8fd00)1080-30p (https://imgbb.com/)

NTSC 24p
1st: 23b0219=>25.491 fps -- good       2nd: 23b0217=>25.586 fps -- bad
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcRM3eq%2Fsmall-ob-zon-8-24p-timer-A-23b0219-25-491fps-good.png&hash=859a6e58a62c905468ba1a6b55314202)1080-24p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FngqUKq%2Fsmall-ob-zon-8-24p-timer-A-23b0217-25-586fps-bad.png&hash=fe47cf02711e9b66f52510834155b0b6)1080-24p (https://imgbb.com/)   

3x Crop_Mode 30p NTSC
1st: 277025d=> Good (fought to write down fps)     2nd:277025b=> Bad
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FeA01XA%2Fsmall-ob-zo-10-3x-Crop-30p-timer-A-277025d-good.png&hash=0a275e6c322bcbad9adbe24073704a3b)3xCrop-30p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FkHbKKq%2Fsmall-ob-zo-10-3x-Crop-30p-timer-A-277025b-bad.png&hash=842f48adc4495686233f86f4292549da)3xCrop-30p (https://imgbb.com/)

PaL 1080 25p
1st: 2570219=>27.881 fps Good    2nd:  2570217=>27.985 fps Bad
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FcOXWXA%2Fsmall-ob-zo-12-25p-27-881fps-timer-A-2570219-good.png&hash=a9c36378e7ded945253a63247b280a5f)1080-25p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FktK6zq%2Fsmall-ob-zo-12-25p-27-985fps-timer-A-2570217-bad.png&hash=f45d6440d167ebd28689685ba3f26c86)1080-25p (https://imgbb.com/)

NTSC 640x480 30p
1st: 23b0219=>31.864 fps Good   2nd:  23b0217=>31.980 fps Bad
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdEs1FV%2Fsmall-ob-zo-14-640x480-30p-31-864fps-timer-A-23b0219-good.png&hash=02829b27984457d58a94092b179e0969)480-30p (https://imgbb.com/) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fjj7JvV%2Fsmall-ob-zo-14-640x480-30p-31-980fps-timer-A-23b0217-bad.png&hash=ff5d75e22e0c75125d8248717e2e4b91)480-30p (https://imgbb.com/)


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on November 21, 2018, 07:43:42 AM
Pretty sure the modified images labeled as "good" are actually bad. Double-check the right side; if in doubt, check the DNG as well. An image is good if and only if all the columns contain valid image data (from the original subject, not noise). It doesn't have to be totally broken to be "bad".

480p is no different from 1080p on this camera; no need to include it.

x5 is only one, regardless of FPS. No need to use FPS override.

The x5 24p (ob-zon-5-3xCrop-24p-over-ride.png) probably was done using FPS override, and looks bad to me. There are 24 bad columns, so timer A - whatever it was in that mode - was wrong by 6 units. That means, FPS override should be fixed as well (FPS_TIMER_A_MIN in fps-engio.c).

In the other images labeled as "good", there are more than 50 bad columns (the chart displays only the last 50), so I'm unable to tell the exact number.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 22, 2018, 05:42:12 AM
Ok ,
I used the "auto Preview" in all the tests , when the image got scrambled when I pushed timer A (decreased the number value)
I would increase the value until I got a clean image , noted the timer-A setting then decrease the number value by "2" this
would scrambled the image , so I called it "bad" .

I'll re-run the tests again with just the ML B/W low res . preview instead of "auto" preview .
with 1080 30p , 24, 25, & 3x crop 30p

Yes I ran 3x crop in 24p with frame over ride enabled .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 22, 2018, 06:40:56 AM
Alright just did 24p in 1080 with ML gray preview ,
Is it right now ? 1080-24p-ob-zones_11-21-2018.rar (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1080-24p-ob-zones_11-21-2018.rar)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FdMMXsA%2Fsmall-ob-zon-1080-24p-Good-timer-A-23b21c-25-349-fps.png&hash=a5e0a0291b7e6ab6f6a25ded5eb3e278)1080 24p Good (https://imgbb.com/)
Code: [Select]
c0f06008 23b021c =>25.349 fps **good**
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2FiPEDzq%2Fsmall-ob-zon-1080-24p-Bad-timer-A-23b21a-25-443-fps.png&hash=879068c7c3cb28834b74beaa7485e15f)1080 24p Bad (https://imgbb.com/)
Code: [Select]
c0f06008 23021a=>25.443 fps **bad**
Basically , I adjusted timer A (increase fps) until I saw a white line/bar in the image on the right side of Liveview and
dialed it back until the white line/bar was gone , (23b021c) that's the one I call "good"

On the one I call "bad" I dialed the value down by 2 (23b021a) .

If this is the right procedure , let me know and I'll finish the rest tomorrow .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on November 22, 2018, 07:14:16 AM
Nope, timer A still seems to be too low.

If in doubt, check or upload the DNG as well (either "dump raw buffer" from raw_diag, or plain silent pictures). Look closely at the last columns (they should not be black or white or otherwise static noise; they should have details from the test scene). These columns may not be obvious on the low-res preview.

Also, please pay attention to parity. Both of these examples show different parities, e.g. Canon value 0x23b is an odd number, modified value 0x21c is even. This causes some non-uniform noise (look closely at the images; it's no longer Gaussian).
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 22, 2018, 07:31:16 AM
Ok , I thought I my have messed up . Now I understand sorry slow learner  :)
Actually I did do a Image_Dump on both tests , just incase there were needed.
Good-RAW-013.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Good-RAW-013.DNG)  & Bad-RAW-014.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Bad-RAW-014.DNG)

I'll re-run the test again & I'll  pay attention to parity
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on November 22, 2018, 08:16:08 AM
These artifacts all over the image are caused by different parity of timer A (compared to Canon value).

From the DNG, I can now do the math. Number of bad columns: 52 in the "good" image, 60 in the "bad" image. Difference of 8. Timer difference: 2 units. One unit of timer A reads out 4 pixels, so it adds up. The "good image" is off by 13 units, the "bad" one is off by 15 units.

The screenshot shows the last 50 columns, and all of them were bad, so I could not do the math from there.

0x21c + 13 = 0x21a + 15 = 553 units. Adding 1 since hardware register is written with timer-1.

Therefore, the minimum timer A value in 1080p is 554 (when doing the math for frame rate) or 553 in the hardware register. Canon default is 0x23b (hardware register) or 572 (when doing the math).

That means, in 1080p24 and 30, we can reduce timer A by no more than 18 units (starting from default value of 572). In 25p, default value will be 600, so you'll be able to reduce it by 46 units.

This value (554) will give the lowest rolling shutter and the highest resolution at any given frame rate. Why not just using this? When aiming for a certain frame rate, timers can be set to integer values, so it may not be possible to get the requested frame rate with 3 decimal places. For example, 23.976 => timer B would be 1807 => 23.974 (closest approximation). For 25p => timer B would be 1733 => 24.998 fps. Or, 1732 => 25.012 fps. For this reason, we should try slightly higher timer A values that may result in frame rates closer to what we expect, but these higher timer A values will require lower values for timer B (which might limit the vertical resolution).

If you'll ever want to reduce horizontal resolution in 1080p (e.g. to get higher frame rates or smaller rolling shutter), you may reduce timer A even more. For example, if you are OK with 1872 pixels, you may use A = 552 (2 units = 8 pixels). If you want higher resolutions, you will need to add 2 units for every 8 pixels; however, on 5D2, you won't be able to increase active area beyond 1880 in 1080p; there are not enough pixels on the sensor. You can still request higher resolutions and get a "4K" frame good for April 1st announcements, if you want :D

Now it's your turn to do the same with x5.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 23, 2018, 03:18:10 AM
Great ! thanks for the math ,

Now it's your turn to do the same with x5.
Ok , Weekend project .

You can still request higher resolutions and get a "4K" frame good for April 1st announcements, if you want :D
Yes , I like that idea ! :))
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 25, 2018, 07:32:04 AM
Continue on with the Timer A experiment , to see how far it can be push in 5x Zoom (3xCrop) .
but before I start I did a 4k frame in 1:1 (FHD) for kicks and
Quote
You can still request higher resolutions and get a "4K" frame good for April 1st announcements, if you want :D

to see what happens , will something I thought that could not work .
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FMGWLJff%2F4104x1249-RAW-015-Small-dng.png&hash=08917c857458de2be332c1860af7c489) (https://imgbb.com/)
1:1-FHD-4104x1249_RAW-015.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1249_RAW-015.DNG)
4104x1249_RAW-015_uncroped.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x1249_RAW-015_uncroped.png)
Code: [Select]
Default Crop Size               : 4104 1249
Active Area                     : 18 160 1267 4264


So I crop off the bottom part & ended up with 4104x594 , as you can see it's vertically compressed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F6ts1crt%2F4104x594-compressed-RAW-015-Small.png&hash=c5a7ff9137988e1c4cb7507855a545d3) (https://imgbb.com/)
4104x594_compressed_RAW-015.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x594_compressed_RAW-015.png)

In Irfanview I vertically stretched the image 300% , pixel binding maybe "1:3" ?

I ended up with 4104x1782  :))
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fph14gK8%2FRAW-015-small-4104x1782.png&hash=ac5602fb63d2a8527b5c3bcdf6b5be35) (https://imgbb.com/)
RAW-015_Unsqueezed_4104x1782.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RAW-015_Uncompressed_4104x1782.png)

This was done in 1:1 FHD not 3x crop(5x zoom) that's what makes this so very interesting
Setting: 9.980 FPS
Code: [Select]
COMS[2] 0x40e ->0xe
c0f06008 0x23b023b ->0x23b080b **not too sure but very close to this**
c0f06088 0x4f40432 -> 0x4f4088a
So when I set CMOS[2] from 0x40e to 0xe that changed the image form the Full HD (pixel binding  & line skipping) to center crop at 1856x1249
then I slowed the frame rate to 9.980 fps and increased the raw image buffer to 0x4f4088a=>4096(4104)
I'm still wonder  how this possible , it shouldn't work unless I stumbled upon pixel binding mode  :o

 Oh yea here the 4k ob zone that started this all
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F7jB5JBR%2F4k-ob-zon-7-small.png&hash=990573799aa6541bf60078aa8f76936f) (https://imgbb.com/)
 4K-FHD-ob-zon-7.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4K-FHD-ob-zon-7.png)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on November 25, 2018, 09:52:22 AM
Yeah, binning factors on X and Y are pretty much*) independent. With the notation from crop_rec, this is the 3x1 binning mode (read 1 line, skip 2, read every column). You probably want to experiment with the opposite, i.e. 1x3.

Binning factor is controlled like this:
- horizontally: from CMOS registers (model-specific, CMOS[2] on 5D2); apparently doing proper binning on all models;
- vertically: from ADTG[1000/100C] or [8000/800C]; one register switches between 5 = read out every line and 6 = enable line skipping/binning, the other selects the number of lines skipped/binned.

*) On 5D3, in this mode (3x1) there are some artifacts I don't know how to fix; register ADTG[8806] appears to control these.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on November 29, 2018, 07:43:11 PM
When i was shooting with this build, i had problem with opening 12 files in mlv producer
I got run-time error 52 - bad file name or number.

Did not have such a problem in the past.

Any ideas?

Tnx
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 30, 2018, 04:42:38 AM
It more then likely a problem with mlv producer , remember there was a issue with too many files open and causing a crash.
Report it on mlv producer thread , and post your log file from mlv producer also try deleting all your "idx" (index files) in the "data" subdirectory folder "IDX"


 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 30, 2018, 04:48:26 AM
Yeah, binning factors on X and Y are pretty much*) independent. With the notation from crop_rec, this is the 3x1 binning mode (read 1 line, skip 2, read every column). You probably want to experiment with the opposite, i.e. 1x3.

Binning factor is controlled like this:
- horizontally: from CMOS registers (model-specific, CMOS[2] on 5D2); apparently doing proper binning on all models;
- vertically: from ADTG[1000/100C] or [8000/800C]; one register switches between 5 = read out every line and 6 = enable line skipping/binning, the other selects the number of lines skipped/binned.

*) On 5D3, in this mode (3x1) there are some artifacts I don't know how to fix; register ADTG[8806] appears to control these.

That's great , I gave 3x1 a try but unsuccessful -- It seem I can't find that long vertical with CMOS[2]
more investigation needed , I must be missing something .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on November 30, 2018, 07:08:09 AM
I'm starting to finalize the crop_rec preset's , I'll keep 2880x1080 as is but it will not have the broken preview
as I found a fix with a1ex's help (as always  :) ) here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208497#msg208497) is the post with proof.

2400x1200 @23.976 is next , It's 90% , just getting some image ghosting a little , but clean black levels .
5x frame blanking & CMOS[1] reg's were needed
CMOS[1] was changed to 0xe6a ->0xa28 (I think , I forgot to write it down)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fxh6Z3yc%2FVRAM2-23-9762-4kx1-2k-almost-clean-small.png&hash=bdd75b323f2acbfad4cd821331085697) (https://imgbb.com/)

2144x1200 @ 23.976 fps is stable and ready to go , post is here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208478#msg208478) with samples
4096x1074 @ 18fps is working , and I will include it in the presets , I may be able to squeeze a few more frames/sec to 20fps
full res 5632x3750 @4 fps
I do plan on including a squeezed 48/50fps 1856x704 but there's still a black level issue , so that need to be fix before I can include it


This is not the final list of crop_rec preset's , but these are the ones I can get to work right now


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on November 30, 2018, 02:16:45 PM
reddeercity,

you were right!
the problem was with mlv producer. I did not use latest version. No problems with latest one.

very excited about your future crop build.

tnx
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on December 01, 2018, 09:21:59 PM
I'm starting to finalize the crop_rec preset's , I'll keep 2880x1080 as is but it will not have the broken preview
as I found a fix with a1ex's help (as always  :) ) here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208497#msg208497) is the post with proof.

2400x1200 @23.976 is next , It's 90% , just getting some image ghosting a little , but clean black levels .
5x frame blanking & CMOS[1] reg's were needed
CMOS[1] was changed to 0xe6a ->0xa28 (I think , I forgot to write it down)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fxh6Z3yc%2FVRAM2-23-9762-4kx1-2k-almost-clean-small.png&hash=bdd75b323f2acbfad4cd821331085697) (https://imgbb.com/)

2144x1200 @ 23.976 fps is stable and ready to go , post is here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208478#msg208478) with samples
4096x1074 @ 18fps is working , and I will include it in the presets , I may be able to squeeze a few more frames/sec to 20fps
full res 5632x3750 @4 fps
I do plan on including a squeezed 48/50fps 1856x704 but there's still a black level issue , so that need to be fix before I can include it


This is not the final list of crop_rec preset's , but these are the ones I can get to work right now




Sounds amazing.

Are any of these available to test yet?

Also do you have a code repository/upload it somewhere?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 02, 2018, 05:06:33 AM
No , not yet , thou you could load the adtg_gui.mo (https://builds.magiclantern.fm/modules.html#iso-research) and manual set it .
I'm not the fastest or best coder  :D so may take me a little time , plus I would like to update 5d2 4k branch in to the main 4k branch
since we have "raw_slurp" working now ( 10bit_12bit (https://builds.magiclantern.fm/experiments.html) FHD) like 5d3/D5 cams , there will be no need for waza57 edmac hack (to extend higher resolutions)

I may also add a cheap & dirty 4k/3.6k squeezed 3x1 in 1:1 FHD , there will be some Aliasing but not as bad as 3x3 .
I'm close to getting 24 fps (currently 20.5 fps unstable ,  17.3 fps stable no image artifacts) @ 3648x660 squeezed ( un-squeezed 3648x1980) 1.85:1 A.R. almost 16x9  :)
I'm most likely will have to reduce the vertical to 2.35 A.R. (1550) (squeezed 519) to get more fps , or more -- still experimenting .

@Ilia3101 , being using your app (mlv app 1.3) to get the right final size with 3x1 , thanks
It's 500% better then the last time I use it , (6 months ago)

From Image_Dump
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FTtfX9D9%2F3672x661-squeezed-RAW-018-small.png&hash=e6b71c5b3dc271d0af0ab0101e6f4a1c) (https://imgbb.com/)
3672x661-squeezed-RAW-018.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3672x661-squeezed-RAW-018.DNG)
3672x661-squeezed_RAW-018.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3672x661-squeezed_RAW-018.png)


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FTmZHkxq%2F3672x1983-RAW-018-small-360.png&hash=c1c4744fb909ed5365b2a8da7fd8d71c) (https://imgbb.com/)
3672x1983-RAW-018.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3672x1983-RAW-018.png)

Here a short 13Mb clip 14 seconds 3648x1980-17.3fps-M01-0004.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3648x1980-17.3fps-M01-0004.mp4)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 02, 2018, 07:58:56 AM
Ok got it !  :)) 24 fps !! 3.6k Squeezed !!!
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F5LkrV1T%2FSqueezed-RAW-020-small.png&hash=992bc0940b9956678bd4402bc44e5d6a) (https://imgbb.com/)

3672x513-RAW-020.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3672x513-RAW-020.DNG)
3672x513-SqueezedRAW-020.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3722x513-SqueezedRAW-020.png)

3672x1539 @ 24 fps A.R. 2.375:1
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FyQfrQbt%2FUn-Squeezed-RAW-020-Small-360.png&hash=e07a13cb7bc476f217bec7371adc7174) (https://imgbb.com/)
3621x1539-UN-Squeezed_RAW-020.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3621x1539-UN-Squeezed_RAW-020.png)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fk07x1JG%2FVRAM11-small.png&hash=96b1c493a35cf3556a93062b3a21cf44) (https://imgbb.com/)


2 clips  -  3648x1536 @ 24.003 fps
No-frame_blanking-M01-2328.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/No-frame_blanking-M01-2328.mp4)
Frame_blanking-M01-2343.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Frame_blanking-M01-2343.mp4)

3672x513 @ 24.004fps
ADTG1[1061] frame blanking was add to try and get the frame ghosting to stop , still need more work.  :-\

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FTHFWhCT%2FVRAM12-small.png&hash=d7e35b8985f954f63a5172ec3572aa5e) (https://imgbb.com/)

A & B Timer info
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FwrvS3s1%2FVRAM10-small.png&hash=144cd296710a859f10360ae036fdfc21) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on December 02, 2018, 08:20:37 PM
wow...cant wait!!!

You are the man.

This is more than a  dream for 5d2
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 03, 2018, 08:38:59 AM
Going back to Hi-Frame rate 50/48fps to see if I can clean up the black levels.
Got side tracked , wanted to see how far I could push the frame rate passed 50 fps
in a smaller resolution (squeezed-1600x500=>un-squeezed-1600x832) (don't think 1856 will work , but didn't try)
I could push it to 64.020 fps but Liveview seemed to lockup/freeze -- battery pull.

But I can record at a little less frame rate 63.050 fps !! :D
Thou 62.5fps is more stable and has at least good black levels (101-1013 10bit) and mostly if not all artifact free.

Setting for 63.051fps @ 1600x500 (squeezed)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fj3wXbT3%2FVRAM18-63fps-small.png&hash=19a99befa0c85a0c801095036075e7ee) (https://imgbb.com/)

From .mlv file @ 62.50fps Squeezed
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FW037mWQ%2FM02-2327-000000-small.png&hash=6d47cb7a71663362e2711ba253a16425) (https://imgbb.com/)
1600x500-M02-2327_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1600x500-M02-2327_000000.dng)
1600x500-M02-2327_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1600x500-M02-2327_000000.png)

Un-Squeezed 1600x832 @ 62.50fps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FMRSgHPC%2F1600x832-M02-2327-000000-small.png&hash=27ddaa91c89b1b7109326e8d4e19c3bf) (https://imgbb.com/)
1600x832-M02-2327_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1600x832-M02-2327_000000.png)

Here a short 5 second h264 clip -- 1600x832 @ 62.50fps
1600x832-62.5fps-M02-2327.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1600x832-62.5fps-M02-2327.mov)

Photo from my iPhone with the camera @ 60.024fps
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FVtXvzJ4%2F60-024fps.png&hash=2569c4435c11eeea041721d615d3c5d1) (https://imgbb.com/)

There one thing really puzzles me , as I increase the frame rate with timer "B" the image clean up ,
the black levels are better and no corruption . If you pan fast to a low light area then back to a Hi-Constat (bright Object) then there's the
odd corrupted frame or frames .

Wondering about sensor readout , 5d2 as noted by a1ex is 100 Mpixel/s (Mega pixel per second) on 4 Channels or 24 per channel
so could the bad black level/corruption be that at lower fps  (48/50) the channel band width is too wide as the 5d3 has twice this (at 8 channels) so
a narrower band width . Not too sure how to read this , I guess this sensor is too slow for what I'm expecting it to do.
Still trying to get my head around this sensor readout stuff to understand the limitation correctly .

Edit: found this about readout on CMOS sensor , Interesting !
https://www.researchgate.net/figure/a-Readout-direction-in-one-channel-and-b-the-timing-for-sensor_fig2_238541020
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on December 03, 2018, 11:10:56 AM
Ok got it !  :)) 24 fps !! 3.6k Squeezed !!!
Could you explain me the reel pro of a high horizontal resolution squeezed at 24fps ? I can understand at 62fps to minimize the data rate per picture and getting better fps.
In this test image the aliasing is awful once unsqueezed, so I'm thinking I'm miss something.
David
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 04, 2018, 01:18:07 AM
Could you explain me the reel pro of a high horizontal resolution squeezed at 24fps ? I can understand at 62fps to minimize the data rate per picture and getting better fps.
In this test image the aliasing is awful once unsqueezed, so I'm thinking I'm miss something.
David

Those are all experiment's for investigation purposes .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 04, 2018, 01:43:11 AM
Ok I think I have found 3x1 pixel binding for reduce aliasing .

1856x1248 3x3 @ 29.97 fps (I haven't increased the raw height yet to 3750)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F19JV6dj%2FM03-1806-00001-3x3-small.png&hash=0953feb60f33b4f78dbbe3413b9efe84) (https://imgbb.com/)
M03-1806_00001-3x3pinning.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M03-1806_00001-3x3pinning.png)

1856x412 3x1 
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FHPBdTMt%2FM03-1806-00001-3x1pinning-small.png&hash=7eae1b3af7cbd0b482af15e8dc0d0612) (https://imgbb.com/)
M03-1806_00001-3x1pinning.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M03-1806_00001-3x1pinning.png)

Code: [Select]
CMOS[1] 0xc00->0xbe0
ADTG12[100c] 0x2->0x100
c0f06084 0x10036 ->0x36 **I think , have to check again**

Need to clean up ,  just my very first try & test with this binning mode , still experimenting .
In theory , 3750/3=>1250
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 05, 2018, 07:10:15 AM
Finding different pining modes on 5d2 in 1:1 FHD (3x3) with
Code: [Select]
ADTG12[100c] 0x2 ->100Compresses the image vertical 3x - maximum raw height 402=>1206
1856x402-M04-2332_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1856x402-M04-2332_000000.png)
1856x1206-300%-M04-2332_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1856x1206-300%25-M04-2332_000000.png)

The image pretty bad , so I won't post a screen shot

Code: [Select]
ADTG12[100c] 0x2 ->107Compresses the image vertical 6x - maximum raw height 210=>1260
1856x210-M04-2336_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1856x210-M04-2336_000000.png)
1856x1260-600%-vertical-M04-2336_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1856x1260-600%25_vertical-M04-2336_000000.png)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FRHYVdk8%2FVRAM19-small.png&hash=e1f97bb851043656ad982fcbb2fc1d9c) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 05, 2018, 09:54:56 AM
Ok looks like I found 3x1 I think , As I thought ADTG12[100c] does a lot of the pinning/line skipping

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FZ1SzY2j%2FVRAM21-small.png&hash=16e7ffca434ad22d953d22a6eb55ae01) (https://imgbb.com/)
So 0x0 put the vertical in stretch mode , it seem 33.3% of the original height put the image in proper aspect ratio again
Code: [Select]
ADTG12[100c] 0x2->0x0
1888x1248 ( I changed c0f060084 0x10036->0x10026=> 1888)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F8YfscBj%2F1888x1248-M05-0138-000000-small.png&hash=c1f8ae4a5c1ed5572104a676183736fc) (https://imgbb.com/)
1888x1248-M05-0138_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1888x1248-M05-0138_000000.png)


Re-sized the vertical (33.3% of original height) to 416
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fbitbucket.org%2Freddeercity%2Fmagic-lantern_10-12bit%2Fdownloads%2F1888x416-33%2525-M05-0138_000000_small.png&hash=a1582b1807cdaf613a181072d0ddf389)
1888x416-M05-0138_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1888x416-M05-0138_000000.png)

Still trying to get the ghosting to stop with cmos[1]
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on December 05, 2018, 02:25:38 PM
Ok looks like I found 3x1 I think , As I thought ADTG12[100c] does a lot of the pinning/line skipping
If it's vertical, you found the 1x3 binning ?  ;) The same as theBilalFahkouri in "Magic Lantern Cinema Camera - Dual ISO without aliasing & without quality loss!" thread ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2018, 04:12:28 AM
It's 3x1 (1856 horizontal is still in 3x pinning/line skipping)  & yes but with out dual ISO
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 06, 2018, 06:50:27 AM
Pushing 1280x720 to 72.053fps  :o
A few artifact but I'm more interested in how far the timers can be pushed , with different reg. configuration .

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FJzkTpcy%2FVRAM26-small.png&hash=da0765f1f8ffda2424a8941885beea0e) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FF4WvT36%2FVRAM25-small.png&hash=f85db26a14d1ca567adc8ccbcf3255b6) (https://imgbb.com/)

A short clip 1280x722-72.05fps-M05-2306.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1280x722-72.05fps-M05-2306.mov) -- h264 26Mb

One thing I notice , when I started to pushing "B"timer over 55fps the back level started to clean up & over 65fps black level seem to stabilize .
It's almost like in multiplies of 30 fps gives good black level , e.g. 60fps

Edit: Here are some short hi-frame rate clips that got me to 72fps , the best one is 67fps .
It seems the sweet spot is around 60-62.5fps , I get this with 1600x832 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208955#msg208955)

1280x722-70.086-fps-M05-2305.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1280x722-70.086-fps-M05-2305.mov)
1280x722-69.094fps-M05-2304.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1280x722-69.094fps-M05-2304.mov)
1280x722-68.034fps-M05-2303.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1280x722-68.034fps-M05-2303.mov)
1280x722-67.006fps-M05-2302.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1280x722-67.006fps-M05-2302.mov)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 07, 2018, 08:42:37 AM
Finally got It 2560x1200 @ 23.976fps with Clean preview while recording raw and say Goodbye to Frozen Preview forever :))

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FyFtc8D9%2F2560x1200-VRAM30-small.png&hash=dae61b8bf13b40174a5cfbff82efd8ef) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fst4ksDt%2F2560x1200-M07-0020-00001-small.png&hash=00e6e21911586295e886108cb89310e0) (https://imgbb.com/)
2560x1200-M07-0020_00001.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2560x1200-M07-0020_00001.png)
2560x1200-M07-0020_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2560x1200-M07-0020_000000.dng)
Code: [Select]
Default Crop Size               : 2568 1200
Active Area                     : 52 160 1252 2728

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FydQHx5k%2Fi-Phone-320-small.png&hash=b59724ac3b6a4c7f9c75fa7406be218e) (https://imgbb.com/)
iPhone video of 5d2 showing un-frozen liveview while recording Clean_preview_recording-IMG_2690.MOV (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Clean_preview_recording-IMG_2690.MOV)
Half shutter works for full color preview in standby & recording raw video -- I show this in the video from my iPhone.


Here a short h264 clip 2560x1200-23.976fps-M07-0020.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2560x1200-23.976fps-M07-0020.mov) 36Mb

One thing the bug me is in the raw image dump I can see corruption line on the right side (like timer "A" pushed to far) but it's not in the Raw .mlv video @ 10bit
The image dump is in 14bit , could that be the issue ? but clean in 10bit , strange .
Exiftool black & white level from 14bit dump
Code: [Select]
Black Level                     : 1027
White Level                     : 16200
10bit dng report correct level
Code: [Select]
Black Level                     : 112
White Level                     : 1013

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F1XpZpsJ%2F2560x1200-RAW-038-small.png&hash=972e957b9b42d5634c2d754ed75fc4e3) (https://imgbb.com/)
2568x1200-RAW-038.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2568x1200-RAW-038.DNG)

Can also be seen in the Liveview_dump from the LCD/422
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FSvn6GHP%2F2560x1200-LV-045-small.png&hash=df31079f37f31157156714be431d95ad) (https://imgbb.com/)
2560x1200LV-045.422 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2560x1200LV-045.422)

 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 07, 2018, 08:55:59 AM
There´s no source code to check? Why not post stuff on bitbucket? Very easy to follow and check into things.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 07, 2018, 09:08:55 AM
It's the same code I pointed out to you before ,  I'm doing everything with ADTG_GUI.mo
so there no special crop_rec module , just plain old mlv_rec in 3x Crop_ mode (5xZoom)

The next step is to code it in to the crop_rec module .
Everything can be reproduce , that why I post of those images so other can fellow along .
It you what to get other D4 cams working with this , first the redirect buffer("raw_slurp") need to be
figured out and I also pointed this out before .

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 07, 2018, 09:14:13 AM
Ok I see. Is the generic power time code in crop_rec.c applied to 5d2? Might fix the stripes if so or at least there would be a place to look for fixing.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 09, 2018, 12:32:11 AM
No , 5d2 doesn't use those. The image is clean in 10bit and that what really counts as 14bit would be only 1-2 second of record time
At least 10bit I can get 15-20 seconds , until I can get working on Lossless & CF interface overclocking .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 09, 2018, 02:21:38 AM
Ok I'm starting to code this (any help/advice would be great) 
I'm not the greatest at this so please be patient :)
All the new valve's I'm using are in the previous post #379
I'm working off crop_rec_4k_5d2 branch , waza57 source

I'm coding a 2560x1200 @ 23.976 preset

crop_rec.c-1436 (https://bitbucket.org/waza57/magic-lantern/src/eff925c034ff006014d2f026bba3f8259884a220/modules/crop_rec/crop_rec.c?at=crop_rec_4k_5D2&fileviewer=file-view-default#crop_rec.c-1436)
Code: [Select]
if (is_5D2)
    {
       int timerA = 727;
       int timerB = 1374;
       int a = reg_override_fps_nocheck(reg, timerA, timerB, old_val);
       if (a) return a;
       
       switch (reg)

crop_rec.c-1684 (https://bitbucket.org/waza57/magic-lantern/src/eff925c034ff006014d2f026bba3f8259884a220/modules/crop_rec/crop_rec.c?at=crop_rec_4k_5D2&fileviewer=file-view-default#crop_rec.c-1684)
Code: [Select]
   for (uint32_t * buf = (uint32_t *) regs[0]; *buf != 0xFFFFFFFF; buf += 2)
    {
        uint32_t reg = *buf;
        uint32_t old = *(buf+1);
        if (is_5D2)
        {
          if (reg == 0xC0F06088) 
            {
               engio_vidmode_ok = (crop_preset == CROP_PRESET_CENTER_Z)
                ? (old == 0x46A04BA)                        /* x5 zoom */
                : (old == 0x4E7058A);   /* 1080p or 720p */
            }
        }
        else

But my question is this :
I need to add
Code: [Select]
c0f0713c
c0f07150
c0fo8184
c0f08188
c0f08518
Do I add them after the raw resolution like this
Code: [Select]
case 0xC0F0713c:
                return (old_val & 0xFFFF0000) + 0x500
or
Code: [Select]
if (reg == 0xC0F0713c) 
            {
              ? (old == 0x476)                     
                : (old == 0x500)
             }
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 09, 2018, 08:52:46 AM
I built a preset from your registers 2560x1200 in here:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec.mo

Wonder if would work with 5D2 at all?

Working in this branch. Didn´t commit any changes yet:
https://bitbucket.org/Dannephoto/magic-lantern/branch/crop_rec_4k_mlv_snd_isogain_1x3_presets

I don´t have the 5D2 so take it with a grain of salt...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 09, 2018, 09:32:07 AM
Ok , can you post your code please
I need to look at the code first before I try it ,
Don't what to brick my camera .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 09, 2018, 09:43:58 AM
code:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec.c
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 09, 2018, 09:48:12 AM
Thanks , I'll have a look later tonight , early here - 2am
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 09, 2018, 09:57:31 AM
If it doesn't work here it should be fairly easy to put it into waza57 code.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 10, 2018, 04:05:06 AM
Yea it's different then the D4 branch , anyways I mainly was looking for the code structure
you gave me some good example to follow , that helps a lot .
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 11, 2018, 09:34:33 AM
Look in to HDMI output Reg's
Code: [Select]
c0f11314
2eb045f->2eb0dof
In Canon Preview
Changes the liveview HDMI HD Width only not raw , I was able to have proper field of view in Crop_Mode 2144x1074 on the HDMI
instead of having a center crop on the hdmi  (still a little distorted & compressed vertically) . Normally with HDMI while in Crop_mode
Let say at 2144x1074 the image gets centered crop to 1920x1080  so you loss the field of view unless you are at 1920 all ready .
I still have to find the HD height yet the first set of valve didn't do anything.
I was mainly only trying to get rid of the black bar on the side of the hdmi ( this cause it to be 1650x1080 and I looking for 1920x1080 or more)
But I think I may have to increase the HD buffer some how , still early in this investigation/development .
I had my Ninja connected to hdmi while I adjusted the reg's and recorded a short 2min clip 85Mb
First I set the reg . then at the 20 second mark I compare it to the reg turn off

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FynvSf5Z%2F5d2-HDMI.png&hash=b11c62a73c188dc9f84f72dfbfabd92a) (https://imgbb.com/)

 H264-5d2-hdmi-00208.MOV (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/H264-5d2-hdmi-00208.MOV)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Aperture Science on December 15, 2018, 05:06:45 AM
Is there any way to boost 5D2's CF slot speed up to 90m/s? Seems like the 4k recording is about to success, but the slot is dragging all the things slow. By changing the speed of the "Bus speed?" (I have no ideal what is in the camera). I am interested in the "cf_acc.mo" could anyone help me explain what should I change in the "cf_acc.mo"? I will test the value and the result (Hopefully I will not brick my camera lol)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 16, 2018, 05:15:28 AM
Yes ,A1ex experimented with this before . But not sure that the total answer -- We are very close to getting Lossless compression working
It a matter of finding the correct "TwoInTwoOut" r/w addresses .
Currently I can capture a lossless FRSP  (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198044#msg198044) (5632x3752) but can't save it to the CF card , cam freezes .
So from my understanding , the module just needs to be compiled -- I looked in to it a few weeks ago but never got around to compiling it
read this here (https://www.magiclantern.fm/forum/index.php?topic=12862.msg206061#msg206061)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 16, 2018, 08:44:31 AM
just a update , fix 2560x1200 @ 23.976 preset , got rid of the corruption on the right side of the 14bit raw_dump image
Pushing this preset , and the max I can get is 2712x1200 @ 23.976 , this close to the limits for vertical resolution @ 23.976
Unless I can get more resolution  @ 23.976 playing around with the Head Timer 3 & 4 . The problem isn't getting higher resolution
it's getting higher frame rate (23.976) instead of 12 or 18 fps @ 4k ( I could use a vertical compression height trick to get 4k)

Almost got 3008x1080 @ 23.976 working  , still have some frame corruption , (frames are moving sideway & out of order)
need to find a better CMOS[1] reg.

Hoping to have a crop_rec module for Xmas or by the New Year for tester's
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 16, 2018, 09:20:45 AM
just a update , fix 2560x1200 @ 23.976 preset , got rid of the corruption on the right side of the 14bit raw_dump image
Pushing this preset , and the max I can get is 2712x1200 @ 23.976 , this close to the limits for vertical resolution @ 23.976
Unless I can get more resolution  @ 23.976 playing around with the Head Timer 3 & 4 . The problem isn't getting higher resolution
it's getting higher frame rate (23.976) instead of 12 or 18 fps @ 4k ( I could use a vertical compression height trick to get 4k)

Almost got 3008x1080 @ 23.976 working  , still have some frame corruption , (frames are moving sideway & out of order)
need to find a better CMOS[1] reg.

Hoping to have a crop_rec module for Xmas or by the New Year for tester's

which register are you setting for these last presets? So if you go abit lower let´s say 3008x1040 it should be corruption free?

Vertical compression height trick? How?


EDIT:
Cool. Got 3096x1320 24 fps on my 100D now.


Code: [Select]
        case 0xC0F06804: return 0x5490331; // 3072x1320  x5 Mode;

        case 0xC0F06824: return 0x3ca;
        case 0xC0F06828: return 0x3ca;
        case 0xC0F0682C: return 0x3ca;
        case 0xC0F06830: return 0x3ca;
       
        case 0xC0F06010: return 0x34b;
        case 0xC0F06008: return 0x34b034b;
        case 0xC0F0600C: return 0x34b034b;

        case 0xC0F06014: return 0x5f7;
        case 0xC0F07150: return 0x428;
        case 0xC0F0713c: return 0x555;
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 16, 2018, 05:37:37 PM
Cool. Got 3096x1320 24 fps on my 100D now.

That's a Digic V camera. Somewhat off topic but good to know this stuff applies to all ML enabled cameras.

What would be sweet is taking reddeercity's findings on the 5D2 and applying it to other Digic IV cameras. IDA_ML has been pushing to get this on the 7D but I keep getting lost trying to figure it out. The 50D might be easier to port -- @aprofiti are you following this?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on December 16, 2018, 07:22:36 PM
That's a Digic V camera. Somewhat off topic but good to know this stuff applies to all ML enabled cameras.

What would be sweet is taking reddeercity's findings on the 5D2 and applying it to other Digic IV cameras. IDA_ML has been pushing to get this on the 7D but I keep getting lost trying to figure it out. The 50D might be easier to port -- @aprofiti are you following this?

Oh yeah, can you imagine having these insane resolutions on the 7D with its 90 MB/s write speed?  3k continuous recording at 24 fps and 10-bit lossless, no aliasing, no VAF filter, no focus pixels, small crop factor, wonderful colors, unique filmic look - that would be a dream come true for all 7D shooters!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on December 17, 2018, 03:47:51 AM
Hi all. First thanks to Reddeercity and other guys for keeping working on it all. I check this thread every day.
I have a question. Maybe someone else has this problem. My 5d2 has a bad feature. When I magnificate the image with the 5x or 10x mode, it jumps back to 1x after a few seconds (from 1 to 30 second approx. It depends...) And if I don’t have time to press the recording mode, the mode changes from crop to standard and the resolution goes down again to 1856 ... very annoying glitch. In theory, is there any program way to force magnification?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 17, 2018, 04:28:00 AM
@Igor_Braun , No I haven't had this issue , have you tried to clear your setting folder ?
Are you using the build from the first page (https://www.magiclantern.fm/forum/index.php?topic=19336.msg182476#msg182476) ?
Or you could make a short cell phone video and post it here , showing this problem .
Are you using the crop_rec 3.5k preset (2880x1080) when this happens ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 17, 2018, 04:34:12 AM
Vertical compression height trick? How?
Look here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208930#msg208930) & also here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208931#msg208931)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 17, 2018, 04:52:28 AM
Oh yeah, can you imagine having these insane resolutions on the 7D with its 90 MB/s write speed? 
3k continuous recording at 24 fps and 10-bit lossless, no aliasing, no VAF filter,
no focus pixels, small crop factor, wonderful colors, unique filmic look - that would be a dream come true for all 7D shooters!
Just to be clear 7D dose not have focus pixels (https://www.magiclantern.fm/forum/index.php?topic=16054.msg155923#msg155923) if you see any that's would be hot pixels.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 17, 2018, 05:24:17 AM
@dfort , the redirect buffer needs to be the very first thing to be figured out , all the rest will fall in place .
Being it's a dual processor , makes it that more difficult .
I do have my eye on a 7d with a battery grip at my local canon camera store in there used/trade in's dept. for $300 cad = $224 usd
If it's still there in the new year , I may buy it to help port this but not to sure .
I'm actually looking in to a use Red Scarlet X (https://www.ebay.ca/itm/Red-Scarlet-X-DSMC-4K-Camera-Kit-with-Accessories/292816232987?hash=item442d35221b:g:ev8AAOSw1IVb52A~:rk:2:pf:0) right now , has ef mount plus everything to get to work .
I'm a big fan of the FCPX Red camera support , fit my pipeline perfectly.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 17, 2018, 08:00:40 AM
Truth be known I've been shooting mostly with a Canon C300 lately, mostly with EF-S lenses that work nicely with the dual pixel autofocus. No pipeline or playback issues with these files.

It seems that one of the most important tools, adtg_gui module, doesn't work on the 7D. I just dug into it (https://www.magiclantern.fm/forum/index.php?topic=10111.msg209402#msg209402). I've got a love/hate relationship with this camera. One day I'm thinking about getting back into the Portable ROM dumper (https://www.magiclantern.fm/forum/index.php?topic=16534.msg192336#msg192336) to figure out how to pry into the master processor so I can properly finish the firmware update to 2.0.6 and the next day I'm looking for someone who will take it off my hands so I can regain my sanity. Depending on my mood I might let you have it for a bottle of Canadian maple syrup.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on December 17, 2018, 11:40:37 AM
@reddeercity, you know, I think this is my camera glitch, because it doesn't depend whether it has Magic Lantern installed or not. I will make a small phone video...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 18, 2018, 05:58:05 AM
Truth be known I've been shooting mostly with a Canon C300 lately, mostly with EF-S lenses that work nicely with the dual pixel autofocus. No pipeline or playback issues with these files.
Nice , the close one to me is 200 mile away , a real pain -- thou they will ship it to me
But they do have a c300 markii $350 day/weekend (fri-monday) or $850 a week , not too badly priced for body rental
If I was doing something serious , I'd be all over that  :)
It seems that one of the most important tools, adtg_gui module, doesn't work on the 7D.
adtg_gui module was broken for d4 cam up until a few mouths ago , I discovery this when I could  made changes with digic poke (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205062#msg205062) and not adtg_gui module .
a1ex fixed adtg_gui here (https://www.magiclantern.fm/forum/index.php?topic=10111.msg205480#msg205480) for d4 cams , as I can confirm for 5d2 , the adtg_gui.mo on the
module download page (https://builds.magiclantern.fm/modules.html) is the one I have been using of late . So it should work on 7d.

I've got a love/hate relationship with this camera.
One day I'm thinking about getting back into the Portable ROM dumper (https://www.magiclantern.fm/forum/index.php?topic=16534.msg192336#msg192336) to figure out how to pry into the master processor so I can properly finish the firmware update to 2.0.6 and the next day
I'm looking for someone who will take it off my hands so I can regain my sanity.
Depending on my mood I might let you have it for a bottle of Canadian maple syrup.
Sure I'll sent you 2 ,  It like water up here  :P 
which do like better  , Grade A: Light Amber, Medium Amber, Dark Amber.  ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 18, 2018, 07:04:47 AM
What branch are you using for compiling for the 5D2? Checking your autoexec.bin it says:
Code: [Select]
Magic Lantern crop_rec-4k-5D2-eXperimental.2018Oct22.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 8bfbb0ca228e+ (crop_rec_4k_5D2) tip
Built on : 2018-10-23 05:49:51 by david@reddeercity

Tried downloading a 7D version from  Branch: lua_fix but it from will not work with adt_gui.mo from iso-research branch.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 18, 2018, 08:38:47 AM
If you turn on the adtg_gui module on a 7D you'll see that it won't load. Take a look at adtg_gui.c you'll see that the code for the 7D is missing. I was able to find the missing pieces (https://www.magiclantern.fm/forum/index.php?topic=10111.msg209402#msg209402) so the module will load but it still doesn't work. Maybe you need to poke the 7D some other place?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 18, 2018, 09:32:50 AM
Forgot to mention i updated 7D with above code before loading it.
I could of course start erasing error codes but it would be the better starting point to get a branch and a blessed  :P adtg_gui module like the one for 5D2.
Then again I am maybe only opening up a can of worms...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on December 18, 2018, 01:20:27 PM
Then again I am maybe only opening up a can of worms...

Ha ha, Danne, there is nothing bad opening a can of worms.  You can catch a big fish using them as a bate.  Without worms you catch nothing!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on December 18, 2018, 06:13:57 PM
I could of course start erasing error codes but it would be the better starting point to get a branch and a blessed  :P adtg_gui module like the one for 5D2.

The current iso-research branch should work on the 5D2.

I can confirm for 5d2 , the adtg_gui.mo on the
module download page (https://builds.magiclantern.fm/modules.html) is the one I have been using of late . So it should work on 7d.

Not on the 7D without this:

modules/adtg_gui.c
Code: [Select]
    else if (is_camera("7D", "2.0.3"))
    {
        ADTG_WRITE_FUNC = 0xFF2C0944; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS_WRITE_FUNC = 0xFF2C0B3C; //"[REG] ############ Start CMOS"
        ENGIO_WRITE_FUNC = 0xFF1F6B20;  // from stubs
        ENG_DRV_OUT_FUNC = 0xFF1F675C;
        SEND_DATA_TO_DFE_FUNC = 0xFF32B800; //"[REG] DFE:[%#lx]"
    }

I'm pretty sure these are the missing pieces but it doesn't work. Maybe double check it?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 19, 2018, 08:52:58 AM
What branch are you using for compiling for the 5D2? Checking your autoexec.bin it says:
Code: [Select]
Magic Lantern crop_rec-4k-5D2-eXperimental.2018Oct22.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 8bfbb0ca228e+ (crop_rec_4k_5D2) tip
Built on : 2018-10-23 05:49:51 by david@reddeercity
Tried downloading a 7D version from  Branch: lua_fix but it from will not work with adt_gui.mo from iso-research branch.
I was explaining to another user , the source link is here with all the bits to work with 10-12bit FHD
https://www.magiclantern.fm/forum/index.php?topic=19336.msg208026#msg208026

Have you tried "Digit Poke" ?
You manually input any reg. number & change it's parameters
https://www.magiclantern.fm/forum/index.php?topic=19336.msg205062#msg205062
Look like 7d not define , for some reason.
will needs to be fix , I can't help much with this as a 7d is needed .
when I catch up on the my coding of the crop_rec module for 5d2
then I can help more .

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on December 20, 2018, 11:41:04 PM
The 50D might be easier to port -- @aprofiti are you following this?
Yes, I kept reading what reddeercity is posting and maybe I can jump doing some tests in the next weeks, but I need some help from you guys.

Looking at register map (http://magiclantern.wikia.com/wiki/Register_Map) to have a better idea of what is going on here, but I feel the need of more documentation about...

From what I understood, first I need to figure out which registers are responsible of height and width of liveview/capture image and then figure out how much fps can be pushed on each resolution, using raw_diag to check bounds for timer values. Is this right?

I tried to poke some registers with adtg_gui in iso-reaserch branch, but no visible effects on image size...
Even changing resolution from raw video menu and log values doesn't seems to show any useful info (well no info at all...)

Modified Cmos[1] and Cmos[2] but nothing changed, leaving these register: C0F06008, C0F06084, C0F06088... after figured out how to increase resolution

Do I need to switch to crop before make anything?
If I remember right I need to push the "zoom" button to switch to 5x, but it doesn't do anything. I think I'm forgetting something about....

Started to read again from the beginning of this thread and I would like to know if is still needed to find/adapt buffer to accommodate larger image size.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 21, 2018, 06:09:21 AM
Experimenting with line skipping & pixel binding (1:1 FHD) with 1x3 to reduce moire and aliasing

Un-stretch 1856x416
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fg6H06ZY%2F1856x416-1x3-small.png&hash=3e8b73f808786fe3d3565758231429b5) (https://imgbb.com/)
1856x416_1x3.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1856x416_1x3.png)

Here the stretch version 1856x1248
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FJnKqH8F%2FM20-1741-000000-small.png&hash=c5a661209a4ee2f57e4af63777590f89) (https://imgbb.com/)
M20-1741_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M20-1741_000000.png)
M20-1741_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M20-1741_000000.dng)

setting
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FWp7jcmf%2FVRAM41-small-a.png&hash=3a5cffaf711b49ee64704cca8bdaea9e) (https://imgbb.com/)

Short 2Mb h264 clip - M20-1741.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M20-1741.mp4)

I don't really see this to be too useful , as it's too narrow (416 h) , if the vertical was 2400 etc.. then maybe , but record time would only be 1-2 seconds
if your lucky , so unless something changes , I won't be continuing with this 1x3 .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 21, 2018, 07:29:28 AM
@aprofiti I'll try explain the basics here ,
1st: need to have a redirect raw buffer , You are in luck  :) because 50D along with the 5D2 now have CONFIG_EDMAC_RAW_SLURP/CONFIG_EDMAC_RAW_PATCH (https://builds.magiclantern.fm/experiments.html)
Why is this important , it all starts with extending the EDmac to have a large raw buffer e.g. =>full res . (10-12bit doesn't hurt either)
read me (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205614#msg205614)
The import reg's are:
Code: [Select]
CMOS[1] = Sensor Vertical (vertical start/stop)
CMOS[2] = Sensor Horizontal (horizontal pos & binning mode)
C0f06008 = FPS timer A (1 unit = 4 pixels horizontally)
C0f06014 = FPS timer B (1 unit = 1 pixel vertically)
C0F06084 = start row/column
C0F06088 = stop row/column (column: 1 unit = 2 pixels)

Rule of thumb , slow down timer "A" to increase horizontally resolution (start around 8-12fps)
All extended resolutions are in 3x Crop_mode (5x zoom) that the staring point
Then increase Timer "B" to bring back frame rate to 24 etc. ....
Code: [Select]
c0f06088 adjusts the raw image size , here a hex to decimal converter is real handy .
Code: [Select]
5d2 FHD default raw size
4f40432
first 3 number "4f4"= vertical resolution
last 3 number "432"=horizontal resolution x2

CMOS[2] 0x10e (for 5d2 crop_mode) horizontal offset up to 3520 .
0xe => up to full width.

Now how do get this to work & extend the resolution ,
Start with the 10-12bit build on the experimental page 50D.109/magiclantern-raw_video_10bit_12bit.2018Oct10.50D109.zip (https://builds.magiclantern.fm/jenkins/job/raw_video_10bit_12bit/51/artifact/platform/50D.109/magiclantern-raw_video_10bit_12bit.2018Oct10.50D109.zip)
then go to modules (https://builds.magiclantern.fm/modules.html)  download page and get adtg_gui/adtg_gui.mo (https://builds.magiclantern.fm/jenkins/job/iso-research/14/artifact/modules/adtg_gui/adtg_gui.mo)  , That's the one I being using of late .

Once you got it all loaded up , you need to activate the ENGIO registers in the advanced tab of adtg_gui ,
then you need to refresh the reg's in adtg to bring up the hidden ones , I usually go to liveview and play a h264  .mov for a few seconds
then brings up the other 900 other registers . Please note this is all experiences from my 5d2 so 50d may have some differences .
Be ready for hundreds of battery pulls  :P
I hope I didn't confuse you , or send you down any rabbit holes .
Make sure you pay attention to any a1ex's post on this thread , actually anything that a1ex posts is all realavent to all cams , though there may be small
little differences , but in the big picture (no pun in tented) all the camera work the same .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 24, 2018, 12:33:00 AM
@aprofiti , Have try it this yet ? I'm courous to see it this work with the 10-12bit build from experimental download page .
If not let me know and I can build it on my 4k 5d2 d4 branch , you may need some of the code from there.
I locally updated waza57 source to include the new raw_slurp patch & redirect  buffer.
You can also download my source code Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018.zip) ,
but you need to add the 50d raw_slurp & redirect buffer code I haven't updated it yet for 50d.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 24, 2018, 06:01:26 AM
Looking like I got a usable 48fps @ 1888x704 (1888x1174)
The black level is off 10bit (127-1013) usually it's (112-1013) but I can clean it up in MLV App 1.3v windows
Have a little issue with MLV App 1.3v -- Hi-Lights ,when pushed turn pinkish
If I set the black level to 78-80 , clean up really nice , I would say almost perfect , I may have too
code in a black level fix , but still a little more work -- need to adjust preview reg's to un-freeze liveview while record raw .

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F6mfm6XF%2FVRAM42-small.png&hash=2697f7230c059713d4f2bb7a6df0642f) (https://imgbb.com/)


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F6ZzCgby%2F1880x1174-48fps.png&hash=0f6b35031438c6194ee0f5351175c0ae) (https://imgbb.com/)
Frame grab from the h264 from MLV App 1.3v win.

Short h264 clip from mlv app 1.3v win
1888x1174_48fps-M23-2150.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1888x1174_48fps-M23-2150.mp4)
1880x704_48fps-M23-2150_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1880x704_48fps-M23-2150_000000.dng)

If any one wants to play with this 48 fps raw clip I loaded it on my google drive , 487 MB
shot @ 800 ISO , shutter 1/59th of second
1880x704_48fps_M23-2150.MLV (https://drive.google.com/file/d/1u-bmXVgluiMQIA__VEgs2mUXVhE60rUw/view?usp=sharing)
Best to use MLV App , set black level to around 78-80
 :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on December 24, 2018, 10:48:06 AM
@aprofiti , Have try it this yet ? I'm courous to see it this work with the 10-12bit build from experimental download page .
If not let me know and I can build it on my 4k 5d2 d4 branch , you may need some of the code from there.
No, not exactly... I tried only to load your linked adtg_gui on top of 10-12bit build, but wasn't working due to linker unable to reference function; probably it's because wasn't compiled for that branch, so I have to retry when I have some spare time.

Crop_rec_4k build could be useful to work with, let me know if you add support for 50D

Edit: Downloaded your source code and applied CONFIG_EDMAC_RAW_PATCH support for 50D. Put in 5x mode and trying to understand what regs are related to resolution.
If I change timer A, image will be altered as expected, but if check resolution from movie tab it doesn't change, even by modifing CMOS regs
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on December 24, 2018, 07:42:32 PM
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F6ZzCgby%2F1880x1174-48fps.png&hash=0f6b35031438c6194ee0f5351175c0ae) (https://imgbb.com/)
Frame grab from the h264 from MLV App 1.3v win.

Short h264 clip from mlv app 1.3v win
1888x1174_48fps-M23-2150.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1888x1174_48fps-M23-2150.mp4)
1880x704_48fps-M23-2150_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1880x704_48fps-M23-2150_000000.dng)


Wow that's a lot of pink moire!!! do you get less with MLVProducer?

What happens if you choose a debayer algorithm other than AMaZE?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 25, 2018, 05:29:53 AM
Wow that's a lot of pink moire!!!
Yea thought so , but remember it's vertically compressed & 5d2 as you know have pixel binding & line skipping
and by compressing it I bet is making the pixel binding worse , I good case for you guys over on the MLVApp1.4 thread
to analyzes and make better algorithm .
What happens if you choose a debayer algorithm other than AMaZE?
It's worse , AMaZE give the best image
do you get less with MLVProducer??
I did a test to see , there a Alias Filter & False Color Filter .
I let the false color filter enabled and toggled off and on the alias filter , mlvproducer has his own
debayer algorithm "MLVPDM" .

MLVProducer-Alias-Filter & False-Colors-Filter enabled
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FMfn7VMY%2FMLVProd-Yes-Alias-Filter-Yes-False-Colors-Filter-1880x704-48fps-M23-2150-00001-small.png&hash=bdac5aed4f9b370c428fd8dd7cc1800d) (https://imgbb.com/)

mlvprod_alias+falsecolor filter_1888x1174_48fps.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVProd-Yes_Alias%20Filter_Yes%20False%20Colors_Filter-1880x704_48fps_M23-2150_00001.png)
h264 clip -- mlvprod_alias+falsecolor filter_1888x1174_48fps.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVProduc-Yes_Alias%20Filter_Yes%20False%20Colors_Filter-1880x704_48fps_M23-2150.mov)

MLVProducer-No Alias Filter & Yes False Colors Filter
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FdQpdkR7%2FMLVProd-No-Alias-Filter-Yes-False-Colors-Filter-1880x704-48fps-M23-2150-00001-small.png&hash=54a9bd9c97f55d094da0555b568bf016) (https://imgbb.com/)

mlvprod_no_alias_filter_yes_false_color_filter.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVProd-No%20Alias%20Filter_Yes%20False%20Colors_Filter-1880x704_48fps_M23-2150_00001.png)
h264 clip mlvprod_no_alias_filter_yes_false_color_filter_1888x1174_48fps.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVProd-No%20Alias%20Filter_Yes%20False%20Colors_Filter_1888x704_48fps_M23-2150.mov)

Don't judge by the small images I posted , there down sized to 16% of the original
I think the it better , what do you guy think ?
Maybe mlvapp 1.4v is better with this , didn't try it yet .

Merry Xmas every one  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 26, 2018, 01:35:46 AM
No, not exactly...
Edit: Downloaded your source code and applied CONFIG_EDMAC_RAW_PATCH support for 50D.
Put in 5x mode and trying to understand what regs are related to resolution.
If I change timer A, image will be altered as expected,
but if check resolution from movie tab it doesn't change, even by modifing CMOS regs
Sorry I may have forgot to mention that
Code: [Select]
c0f06088Is the number 1 reg you need to be concern about mostly , it's what set the Raw Resolution
the rest reg's adjust image , e.g. cmos[1] centers the image etc. ...

Can you post a screen shot of the 50d's "capturing Reg's in agtd_gui ?
I'm mostly interested in
Code: [Select]
c0f06088 values , let start there .

Just as a refresher ,
this is what I'm looking for (in the show capture reg's)
but with the 50D valves as this from my 5D2 in default FHD 1:1 29.97fps

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FJ2FQKM9%2FVRAM43-small.png&hash=ed996239b14237c4d9f252c7817fd35b) (https://imgbb.com/)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FZGsdZ56%2FVRAM44-small.png&hash=3a3e6a71bda06f589a103c448ffed547) (https://imgbb.com/)

If there are not showing up ,
make sure you enable "ENGIO Registers" in the advanced tab
refresh liveview ( play a h264 .mov) should see  around 900 reg's
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FWphWCZV%2FVRAM46-crop-small.png&hash=dd2cd257d41c991c9dd474ff57ee84f1) (https://imgbb.com/)

And when this successful you should see Memory Patches like this , I have 7
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FTtYpV9m%2FVRAM45-small.png&hash=59fc6e87912dc2ad1199a3fa94a4ee93) (https://imgbb.com/)

does this help ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on December 26, 2018, 10:26:21 PM
Yesterday I tried to understand what happen if I poke some regs, this is what I get in 5x mode with default settings:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fh86ywWx%2FVRAM25.jpg&hash=8f9e5ca5525c6e3fcb29f4d6c2e1e639) (https://ibb.co/h86ywWx) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FfYLmFq6%2FVRAM21.jpg&hash=4ca7fa9de76a2462096738e14b8db3f8) (https://ibb.co/fYLmFq6) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FhmTJsFq%2FVRAM22.jpg&hash=3b361e7915b2f40b6ad699815c1bf90e) (https://ibb.co/hmTJsFq) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fk8j1B5P%2FVRAM23.jpg&hash=74eaebeac3d0549d2f3fb4f59b6db99b) (https://ibb.co/k8j1B5P) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FNSgLJMr%2FVRAM26.jpg&hash=d7b4a10ba342ed57c0f2fbe026571789) (https://ibb.co/NSgLJMr) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FtMNJrRB%2FVRAM24.jpg&hash=ab4cb0585f5e483db03ede874aa20d4f) (https://ibb.co/tMNJrRB)

Let's compare behaviour of 50d with 5d2:

If I increase Timer A it will lower FPS, if I decrease value I get more FPS, similar behaviour with TIMER B.

If I try to change image size using c0f06088, it allow me to increase by 1 (value => 0x4530452) and get 1080 vertical but it get upper bounded; tried to lower FPS value (10.572) changing TIMER A => 0x27b070b but no increase in resolution.
Instead if I try to change horizontal resolution (increasing or decreasing) I get distorted pink images...

I saw you modified also other regs (ADTG12[100c], c0f07XXX...) from the one you listed a couple of post above.
How did you find they were necessary? Did you randomly tried register or did you observed what register were changed by canon firmware using adtg_gui?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 27, 2018, 07:43:23 AM
Thanks for the screen shots , but something when wrong I was looking for just the capture size reg's

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FBw3bdwm%2FVRAM49-small.png&hash=278648713bdd5b963d757e0152dda913) (https://imgbb.com/)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FR9M8q82%2FVRAM47-small.png&hash=47ee99ba803368a3b2fee96659ca334c) (https://imgbb.com/)

You should see the same list of reg's I posted not just cmos reg's  , that tells me either
didn't enabled the "ENGIO Reg's" in the advanced tab of adtg_gui.mo or you didn't refresh liveview
e.g. load a h264 of review a photo , this a must or nothing will work and you will not get access to the hidden reg's .

There over 900 reg's you can see when it right but if not there only around 16 or 18 reg's visible
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FfS4KdYf%2FVRAM48-small.png&hash=5dfcef8147dd1c84f91324eb792b8851) (https://imgbb.com/)

The other think that help's with development is the dump_image_buffer
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FP1xpk38%2FVRAM50-small.png&hash=7532a0d0357ea1416b137eab52b0d4aa) (https://imgbb.com/)

This help to analyze the raw image for true raw size before cropping with exiftool

There was 3 reg's that I saw that where useful
Code: [Select]
c0f06088
c0f06008
c0f06014
First let understand what going on here
let look at reg c0f06088 =>0x4520452
What does this tell us ?
Will for starter , this is the raw resolution and we read this in decimal , so let spilt the number up .
452=>1106 that the vertical res & the back half is horizontal 452=> 1106x2 =>2212 -160 (crop off boarders) =>2052
that should close to the Image dump .

Next Timer A =>c0f06008 & Timer B =>c0f06014
So timer A =>0x27b27b , we only use the last 3 (27b) ,
Let do some math , 27b hex = 635 decimal x4 => 2540
So this tell us that the max horizontal resolution that Timer"A" can do , including black boarder crop off area offset etc. ...
On the 5d2 raw image dump exiftool tell me the crop off area or offset  e.g. 52 /160 (52=vertical  & 160=horizontal) in crop_mode

Timer B =>c0f06014=>0x5e5 =1509 ,  max vertical resolution
So a Image dump in crop_mode would be very useful right now to see the offset , not sure if it's the same as 5d2

So if I was you , I would start with a small horizontal increase , from 1920 -- theoretically you shouldn't have to mess with timers
unless you exceed 2540 H , thou the very first thing I would do is slow timers A & B to get 23.976fps of very close to that (24).

This what I do on 5D2 , start @ 29.97fps -- reduce timer A by only a few units (0x2270227=>0x227027b) Increase timer B (0x4ff=>0x59f)=>23.976fps
then increase vertical to 1200 (my case) then increase horizontal in very small increments to 2400 H stopping between changes to check liveview .

The best way to approach this in my opinion , is to understand what each reg does in relation to raw size & preview size.

Also Liveview will messed up or freeze something pink , that canon liveview everything has to done with ML B/W (Gray) Preview.
A half shutter button press should give clean preview plus that how I refresh liveview between resolution changes .

Sorry for the long post and If this seem too simplistic , I'm writing this not only for you @aprofiti  but for everyone else , so it can be understood by all
and get more people to develop .

I saw you modified also other regs (ADTG12[100c], c0f07XXX...) from the one you listed a couple of post above.
How did you find they were necessary?
That was found or discovered by a1ex from what I understand
ADTG12[100c] squeezes the image vertically , that's the 720p 50/60p mode on other cams ,
yes that right 5d2 can do 720p 50/60 all the way up to 72fps (that's where I stop don't push it any more but there was still room to go)
The how I got 1888x704 @ 48fps -- unsqueezed (1.66) I get 1888x1174 @48fps .
c0f0713c & c0f007150 open up the vertical window , it a must .
Explained here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205614#msg205614) , I do belive I posted this before for you to read -- It's a1ex explaining every thing you what to know .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on December 27, 2018, 08:30:50 PM
Yea thought so , but remember it's vertically compressed & 5d2 as you know have pixel binding & line skipping
and by compressing it I bet is making the pixel binding worse , I good case for you guys over on the MLVApp1.4 thread
to analyzes and make better algorithm .It's worse , AMaZE give the best imageI did a test to see , there a Alias Filter & False Color Filter .
I let the false color filter enabled and toggled off and on the alias filter , ...

I think the it better , what do you guy think ?
Maybe mlvapp 1.4v is better with this , didn't try it yet .

Great progress @reddeercity! 48fps on a 5D2 is great!
With MLVApp v1.4 we introduced "Hue vs. Saturation" elements. That means not, that moiree will get any better, but maybe you can make it a bit more invisible by desaturating "a small pink peak". Sure - if you have much (real) pink in the clip you're lost... ;)

Edit: whoa... not only a peak... nearly all pink tones... and still not great.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FXVqqCgf%2FBildschirmfoto-2018-12-27-um-20-41-57.png&hash=f7e0ab7edab8af121ec7ad6449496046)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 28, 2018, 04:20:10 AM
Not too bad , The 1.4v looking good , I gave it a try and compared it to MLVProducer & After Effect -- though I corrected black level
and un-squeezed in MLVApp 1.4v then exported as a Uncompressed Cdng for After Effects .

So from left to right:  After Effect CS6 , MLVProducer , MLVApp 1.4v
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FcyQCnSS%2FA-E-MLVPro-MLVApp1-4v-small.png&hash=2bb5d8fe9f48f7bf0b25e0feb11d8161) (https://ibb.co/Twtb3xx)
A.E.-MLVPro-MLVApp1.4v.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/A.E.-MLVPro-MLVApp1.4v.png)

Here are all 3 PNG's from the above image screen shot
MLVProducer (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVProd-Yes_Alias%20Filter_Yes%20False%20Colors_Filter-1880x704_48fps_M23-2150_00001.png)
AfterEffects_CS6 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/AfterEffects_CS6_1880x704_48fps_M23-2150_00000.png)
MLVApp1.4v (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1880x704_48fps_M23-2150_000000_MLVApp1.4v.png)

Here's the h264 from though 3 App's
MLVProducer_h264 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVProduc-Yes_Alias%20Filter_Yes%20False%20Colors_Filter-1880x704_48fps_M23-2150.mov)
MLVApp1.4v_h264 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/MLVApp1.4v_1880x704_48fps_M23-2150.mp4)
After.Effects.CS6_h264 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/AfterEffects_CS6-1880x704_48fps_M23-2150.mp4)
A.E. rendered out the file as 50fps where the other 2 keep the correct frame rate of 48fps

I tried to grade it as a flat film/log like image across all 3 App's
A.E. seem to be the worst on the test image (lost must of the detail in the center plus data missing/holes)
but the rest of the image is killer thou .
MLVProducer seemed to retain all the detail with very good motion blur , but with pink hi-lights in the almost blowout
hi-lights  , the rest of the image looks similar to A.E. in quality , clean shadows (no noise)
MLVApp 1.4v , I used the Smooth Aliasing filter set to "Last Resort"
Lost detail in the center of the test photo image , has less pink moiré then mlvproducer but more then A.E.
but didn't loose as much detail as A.E. with better motion then A.E. but not as good as mlvproducer , thou it very close.

So I would really be happy with either MLVApp 1.4v Or MLVProducer with a small lean towards mlvProducer .
As far as A.E. is concerned , It was the least desirable image with aliasing & moiré patterns .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on December 28, 2018, 12:57:19 PM
it's not a good workflow, but for curiosity's sake, you might try rawtherapee.

the latest version is quite powerful with dual demosaicing capability.

it would be an interesting test, even just on a single frame...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on December 28, 2018, 03:49:45 PM
@reddeercity
Is it correct to say you are working a 48fps release, what will that resolution be and is it Squeeze or Unsqueezed? Also is the most stable 10/12bit the Oct10 release in MagicLantern Experimental Nightly Build?

@70MM13
RawTherapee sound interesting, I use MLV App but I'm using Camera Raw in AfterEffects the most. I just installed RawTherapee, what is the workflow you use with Magic Lantern? I'm filming a Feature Film exclusively with Magic Lantern, I want the best option to process Magic Lantern footage,  so my Question is what's you favorite stable firmware, and if you were risky and brave to film a feature with Magic Lantern what would be your workflow? Please anyone feel free to answer above questions, I'm very competent but I love to have options and keep my options open so every time I ask a question on this forum its for real productivity. Sometimes I ask question on this forum then it gets loss in the clouds and ignore. Thanks for Any Advice or Guidance
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 29, 2018, 08:00:38 AM
Can 5D2 be set to mv720p 50 or 60 fps? If so it should be possible to set 100c to 3x3 binning and give mv1080p 50 or 60 fps with a little reduced height.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 29, 2018, 08:48:56 AM
There's no real 720p 50/60 mode on 5d2 , it's come about though reg's , actually a1ex figured it out
I'm just fine tuning it ,It's really FHD with squeezed reduced height 1888x704 @48fps I can push it to 750 so it's 3/5
I can do 3x3 with hi-frame rate 72+ @ 1280x720 (raw size) not vertical squeezed.
I did the squeezed version to get full height that's centered. Never really tried to do a un-squeezed FHD at reduced height
I don't think I'll get 48fps more like 40 if that , the sensor it too slow I think for that 5d2 is only 96MP/s where the 5d3 is 192MP/s
and that's the big difference between D4 & D5 . The 5d3 has 8 channels off the sensor where the 5d2 has only 4 channels .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 29, 2018, 09:48:29 AM
So how about 10-12bit 4096x1770 @ 23.976 fps (A.R. 2.31) in 1:1 FHD sound ?
I got a clean 4096x590(squeezed 3x) version I'm thinking of add to the crop_rec
I can record 10bit continuously @ 69MB/s  :D and MLVApp can process then easily
There still a little aliasing , but not that bad you really have to push it to produce it.
Never notice any moiré pattern at all , In this mode the sensor is 1:1 pixel horizontally hence 4096
and the vertical is line skipping every third line from what I can see so 1x3.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FSV644HY%2FVRAM58-small.png&hash=2795033828bfc66a7e9eb3724a5a35c7) (https://imgbb.com/)(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FSfxWnwV%2FVRAM57-small.png&hash=f2fa543437d54f3949cdd668be33ed23) (https://imgbb.com/)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FjwVs6jJ%2F4096x1770-M29-0050-frame-1-small.png&hash=caab8c03264794ba90e887c3aecd01c9) (https://imgbb.com/)
4096x1770-M29-0050_frame.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1770-M29-0050_frame_1.png)
4096x1770-M29-0050_frame_1.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1770-M29-0050_frame_1.dng) un-sqeezed from mlvapp 1.4v
4104x591-RAW-033.DNG (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4104x591-RAW-033.DNG) squeezed , from the image dump
h264 clip 4096x1770-M29-0050.mov (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1770-M29-0050.mov) 70Mb

Pretty much works flawlessly , not issue at all I can see only thing I don't like is there more rolling shutter then I would like .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on December 29, 2018, 10:50:01 AM
Hell Fkin' Yeaaaaah David! :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on December 29, 2018, 02:15:36 PM
...
So I would really be happy with either MLVApp 1.4v Or MLVProducer with a small lean towards mlvProducer .
As far as A.E. is concerned , It was the least desirable image with aliasing & moiré patterns .
I again played a bit around with some freestyle code and your 48fps MLV. The pink in on the chart looks like the green channel is clipped from the low side, not from the high side. So I implemented just for testing a stupid algorithm which reconstructs this part of the green channel. Then I switched to IGV debayer and this is what you'll get then (+ Sharpen=50 to see even more detail):
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FxGcwtt2%2Fshort1880x704-48fps-M23-2150-frame-1.png&hash=be79865aaef5f94829cc76df66f5a837)
The problem is: this is not at all scientific, so we'll kill some areas color with that...

Code raw_processing.c (first the old highlight reconstruction, second the "lowlight reconstruction"):
Code: [Select]
...
                /* Check if its the highest green value possible */
                if (tmp1 == processing->highest_green)
                {
                    pix[1] = (pix[0] + pix[2]) / 2;
                }
                /* reconstruct low green */
                if (tmp1 < 15000)
                {
                    if( pix[1] < 1.1*pix[0] && pix[1] < pix[2] )
                    {
                        pix[1] = (pix[0] + pix[2]) / 2;
                    }
                }
...


So how about 10-12bit 4096x1770 @ 23.976 fps (A.R. 2.31) in 1:1 FHD sound ?
Great!  8) Wanna have! ;)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on December 29, 2018, 06:00:37 PM
simply amazing!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on December 29, 2018, 06:03:19 PM
i mean simply amazing - for 4096x1770 @ 23.976 fps....
just amazing
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on December 31, 2018, 03:05:01 AM
Thanks , did a short clip test in a real world situation (outdoors , shot a skyline @ sunset) .
Didn't like the artifact and saw some aliasing that look not good .
Timers need to be reworked to reduce artifacts before I could use it .
So what did I do to get this , I basically stop the vertical columns (every third one)
Code: [Select]
CMOS[1] 0x40=>0xeif you do the math 5632/3=1877.333 so round it off the same
as the silent picture or the liveview raw image dump 1880x1248 . By increasing the horizontal to 4096 I still have line skipping
vertically (every 3rd line) , but if this was not so I wouldn't be able to record 4k squeezed . So we can call this 4K with trickery
Or 4K line skipping vertically , need to think able this some more .

On the other hand seeing how far I can push Hi-Frame rate squeezed -1280x432 (1280x720 un-squeezed).
Seems I can get to 83.815fps pretty cleanly , thou I can push it to 85.881fps but the image break ups badly , (image moves sideways) 
More then likely if I reduce horizontal from 1280 to less then 1000 I'm sure to get more but this doesn't interest me below 1280x720
So I should be able to give the 5d2 82.5fps safely up to 83.88fps   



Edit: Update -- I push the 4k compressed vertical to 29.970 FPS ,
So that's 4096x590 @29.97 (Un-compressed 4096x1776 @ 29.97 fps)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fpnv8vPG%2FVRAM0-4096x591-29-97fps-small.png&hash=cc4e8d058a6da4576f729a2f2f341d27) (https://imgbb.com/)

Unfortunately it's not continuous at 10bit 29.97fps , I can get around 5000 frames (2.5min) where at 23.976 is continuous .
So that's a plus  :) , it's making this more attractive (even with it's flaws) a little slow-mo in 4k even if it's line skipping
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on December 31, 2018, 05:20:18 PM
Got a 4k squeezed preset working for eosm. Good work RDC:
https://www.magiclantern.fm/forum/index.php?topic=9741.msg209967#msg209967
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on December 31, 2018, 07:48:22 PM
Reddeercity, you do amazing things... look, after I wrote you about the zoom (magnification) jumping out automaticaly, it stopped to do so. Now it works properly. I actually don't know what happened (mystic). So, everything is fine now, but it have been jumping out for almost 7 years till that moment... hm.... So if it happen again I'll show you. Thanks
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 01, 2019, 02:12:31 AM
I can do 3x3 with hi-frame rate 72+ @ 1280x720 (raw size) not vertical squeezed.

You can do 1280x720 unsqueezed, but can you not increase horizontal resolution? I think it's worth it even if vertical resolution or framerate has to be reduced. An 1880x700 48fps preset would be really good.

Also does anyone else with a 5D2 find that when shooting in 2880x1080 the rolling shutter is really extreme?

It is not possible to get usable footage handheld at all. I'm asking because my camera is together by only one screw after repair, so it may just be that.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 01, 2019, 06:24:17 AM
I check in the Frame over ride menu in the advance tab and compared to the other mode ,
Normal crop_mode at 23.976 & 30 plus in FHD at 23.976 (frame override)

2880x1080 crop preset  33.5 -- rolling shutter
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FSBg68ys%2FVRAM17-small-crop-preset3-5k.png&hash=8603170ff73afdf6d2150fafe4aa2f32) (https://imgbb.com/)

Normal 3x crop_mode 2144x1074 @ 23.976 frame override -- 25.6 rolling shutter
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FMV7jXwC%2FVRAM16-small-crop-mode-24-over-ride.png&hash=cced0b5539882d5285b79f62473514ff) (https://imgbb.com/)

FHD 1:1 1856x1044 @ 23.976 frame override -- 23.8 rolling shutter (best)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FTmnRGzq%2FVRAM15-small-FHD-24-over-ride.png&hash=608d7154c2cf9e0ba5ab739a401cb6e8) (https://imgbb.com/)

So yes this more rolling shutter ,
I've being trying to reduce this in normal FHD (1856x1044) I have rolling shutter down to 22.8
Looking to be around 15 but sensor too slow I bet .

FHD 1:1 1856x1044 Low rolling shutter
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FpyZc5Sr%2FVRAM52-small-FHD-22-8-rolling-shutter.png&hash=be0e77ae0346532e71712eca553f348d) (https://imgbb.com/)

Have you try a higher shutter speed  e.g. 1/60th , or  1/90 this should help a little (never tried)
the reason I say 1/60th because timer A is set to 31fps  and timer B just slows it down to 23.976
This is needed to extend the resolution , even in 4k line skipping  (4096x590) to get 23.976 timer A
was pushed pretty high (58 fps) then timer B reduce it to 23.976 .

Unless the timer over head can be reduced (remember a1ex did some test) (https://www.magiclantern.fm/forum/index.php?topic=23040.msg208479#msg208479) he was able to reduce more rolling shutter and here (https://www.magiclantern.fm/forum/index.php?topic=19300.msg207723#msg207723) too
I think on the 5d3 the rolling shutter is around 18 I think .
So I hope to have less rolling shutter , but that work in process .
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 01, 2019, 08:37:12 AM
Pushing to full width (5632) with the vertical compressed so 5632x416 @ 23.99fps = 5632x1248 @23.99fps
Works good , but the full width I can only get 1248 vertical which give an A.R. of 4.5:1  :o
If you are wondering why I can't get a higher vertical resolution , from what I can see the width is
the main factor , as I reduce the horizontal raw size e.g. 4096 I get increased vertical e.g. 1776 etc. ...
So the width & height is in direct proportional to it's self , if I was in 3x crop_mode I wouldn't have this limit .

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fxf7q2gW%2FM01-0022-frame-224-small.png&hash=c60aa6ebdf209c0107f973c8634c9770) (https://imgbb.com/)
M01-0022_frame_224.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M01-0022_frame_224.png)
5632x1248_23.99fps_M01-0024.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5632x1248_23.99fps_M01-0024.mp4)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F9sWGF2h%2FVRAM2-small-5632x416.png&hash=24b103ad3f4a7329f8a457c66d088a09) (https://imgbb.com/)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FqFFg3VL%2FVRAM1-small-full-width.png&hash=4e8c5a7c7368df42a11aa2c2f9f483f4) (https://imgbb.com/)  (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FFBBL5rq%2FVRAM3-small-frame-override-full-width-24.png&hash=a6e690243824dd6c7e1085560f3e4844) (https://imgbb.com/)

It's just tooooo narrow , I would have to get at least 1877 (3:1 A.R.) or 2440 (2.30:1 A.R.) vertical .
So I'll just keep 4096x590 (4096x1776) and develop it further  That seems to be the Sweet Spot for 5d2  :))

Happy NewYear everyone !!
It should be only a few more weeks before my crop_rec module is ready for testing .

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on January 01, 2019, 10:03:21 AM
Unless the timer over head can be reduced (remember a1ex did some test) (https://www.magiclantern.fm/forum/index.php?topic=23040.msg208479#msg208479) he was able to reduce more rolling shutter and here (https://www.magiclantern.fm/forum/index.php?topic=19300.msg207723#msg207723) too
I think on the 5d3 the rolling shutter is around 18 I think .
So I hope to have less rolling shutter , but that work in process .

In 1080p 3x3, the minimum achievable rolling shutter would be...

the minimum timer A value in 1080p is 554

... 554 / 24 MHz = 23.08 μs/line.

You can already get this with FPS override, by manually changing timer A manually to 554 (in the Advanced menu).

Values without FPS override (i.e. with stock Canon firmware):
- 1080p24/30: 576 / 24 MHz = 23.8 μs/line
- 1080p25: 600 / 24 MHz = 25 μs/line

Theoretical speed: 5D2 reads out 4 columns at a time (i.e. 4 analog channels), with a pixel clock of 24 MHz; that would give 96 MPix/second. You just can't get more than that without overclocking (no idea whether this frequency is adjustable by software or not).

That means, the absolute minimum, assuming one can find a way to somehow get rid of the horizontal overheads (which is likely impossible), would be 1880 / 4 / 24 MHz = 19.58 μs/line. Keeping the left OB area (C0F06084: 0x10036,  C0F06088: 0x4f40432) would require 0x432 * 2 / 4 / 24 MHz = 22.38 μs/line.

The remaining (unexplained) overhead is a whopping 23.08 - 22.38 = 0.7 μs/line. Even if that can be reduced somehow, I'm pretty sure it's not worth the effort. You simply won't get significant rolling shutter improvements without overclocking the sensor (or without reducing the horizontal resolution).

1100D, on the other hand, is an interesting beast:

720p25: A = 1000, B = 1280, readout size 1496 x 967 (this includes black borders)
720p30: A = 960, B = 1112, same readout size
Timer A can be pushed to 872 in both modes.

It runs at 32 MHz x 4 channels (i.e. pixel clock is the same 700D & co.), but for some reason, it has a huge overhead. In regular (24/25/30p) movie mode, C0F06084/88 is set to 0x200c3/0x3c903af, i.e. xmax = 1886 (likely rounded to 1888), out of which 390 pixels are cropped from the left side. ML still has to crop 68 pixels of left OB; that gives 1428 active horizontal pixels, 458 (!) for OB width and 2 pixels for rounding (guess); these require "only" 14.75 μs/line. The unexplained overhead would be 12.5 μs/line, or exactly 400 clock ticks. This one is worth fine-tuning, if you ask me.

Why?
- 720p24/30: 960 / 32 = 30 μs/line
- 720p25: 1000 / 32 = 31.25 μs/line
- FPS override: 872 / 32 = 27.25 μs/line
- theoretical value: (872 - 400) / 32 = 14.75 μs/line! (without altering OB areas)

For a 16:9 frame, which would be just 1428x804, it would require only 11.86 ms!

Assuming one can somehow drop 400 pixels from the left OB, that would free another 100 clock ticks, i.e. 11.63 μs/line, or 9.35 ms for a 16:9 LiveView frame.

5D3 also operates at 24 MHz, but has 8 channels. In 1080p 3x3, timer A values are (see here (https://www.magiclantern.fm/forum/index.php?topic=21379.0)):

- 24/30p: 440
- 25p: 480
- minimum value: 398

That gives 18.3 and 20 μs/line with Canon firmware, and 16.58 μs/line with FPS override.

C0F06800/C0F06804 are set to 0x10017/0x528011b; that would require 0x11b * 8 / 8 / 24 MHz = 11.8 μs/line (unexplained overhead: 4.8 μs/line, i.e. 115 clock ticks). Optical black area: Canon crops away 0x17 * 8 = 184 pixels, ML skips 146 + 2, active area width is 1932, total 2264 = 0x11b * 8.

That way, should we find out how to remove that overhead, a 16:9 frame (1920x1080) would require only 12.75 ms.

If we could also get rid of 300 OB pixels (out of 332), that would cut about 1.56 microseconds per line.

Of course, these are just theoretical values; I don't know how to reduce these overheads, I just tried to estimate whether it's worth trying to figure them out, or not. On 5D2, the potential improvement would be very small (the sensor is already used at close to its full speed in LiveView). On 5D3 and 1100D... it's worth taking a closer look.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 02, 2019, 05:32:54 AM
Thanks, much clearer now I'm starting understand this . Yes I see where the 5d3 would be good there.
Now I thinking about 5D4 since this  (https://www.magiclantern.fm/forum/index.php?topic=17695.msg209985#msg209985) happened  :D
I'm keeping my eye's open for a good deal on one , starting to see a few used ones out for sale now .
After I'm done with the 5d2 development , I looking forward to the 5d4.

Now about the 5d2 , I got a new preset UHD Vertical Compressed 3840x624 @ 23.976 (3840x1872 2.0:1 A.R.)
I'm likening this a little more then 4k for the better vertical height , it's a hard to get continuous @ 10bit (more height)
I will keep both presets (3840 & 4096) , the rest will be from the 3x crop mode .

UHD 3840x1872
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FdmNMwMZ%2FM01-2117-frame-170-3840x1872-small.png&hash=dbade20c6d578566f6edf4d51b549dff) (https://imgbb.com/)

3848x625_RAW-040.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3848x625_RAW-040.png)
M01-2117_frame_170_3840x1872_.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M01-2117_frame_170_3840x1872_.png)
3840x1872_23.976_400ISO_M01-2117.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3840x1872_23.976_400ISO_M01-2117.mp4)
3840x1872_23.976_800ISO_M01-2122.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3840x1872_23.976_800ISO_M01-2122.mp4)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FzHbGRcc%2FVRAM5-small.png&hash=dce3c1efdc0b4e1150c48a430f541dbc) (https://imgbb.com/)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on January 03, 2019, 03:11:56 PM
Can't wait... =))

"After I'm done with the 5d2 development , I looking forward to the 5d4."
This also sounds inspiring.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on January 03, 2019, 11:56:03 PM
You should see the same list of reg's I posted not just cmos reg's  , that tells me either
didn't enabled the "ENGIO Reg's" in the advanced tab of adtg_gui.mo or you didn't refresh liveview
e.g. load a h264 of review a photo , this a must or nothing will work and you will not get access to the hidden reg's
Sorry, forgot to say that 50D doesn't have movie playback.
Already enabled "ENGIO Regs" and apparently I can find all relevant regs including adtg12[100c] that were mentioned here, just need to switch category to avoid headache after switching to LV end 3X mode :)

Anyway after some days of swpping batteries (damn LV drawn them quickly...) and trying to match full width, I'm still stuck on solving some problems:
1. There is a pink bar at the bottom of the frame when I increase reg 0xc0f06088 (more resolution, more pink area), no matther how TImer A is modified
2. Increasing width will make room for rigth side of framebut will result also in black bar... In other words image is not centered
    Expected to fix with CMOS[2] but no apparently usefull (Need to redo some testing about) and with 0xc0f06084 (small diff from default appears so leave room for left data but then increasing more will only result in moving image to rigth)
3. a couple of gray/green vertical line in the second half of the frame

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fg60K3Pd%2FRAW-072.jpg&hash=eb026a2e4749a814d609ecfae5ff8266) (https://ibb.co/g60K3Pd) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FLkLDbQ1%2FRAW-073.jpg&hash=72b4ebe40833cae8d063582b72b81952) (https://ibb.co/LkLDbQ1)

First DNG (https://drive.google.com/file/d/1N2xFoEHiVTGkQUSBl7NXSth6CjO37mrd/view?usp=sharing) is default ML preset, second DNG (https://drive.google.com/file/d/12_yolOKlVtt_OWxzoDFNj1nh3QzQU0bo/view?usp=sharing) is attempt of full width:
Code: [Select]
0xc0f06008 : 0x27b04f5 (from 0x27b27b)
0xc0f06084 : 0x1004b (from 0x1004a)
0xc0f06088 : 0x45209bb (from 0x450452)

Not completely sure about photo mode register's values for FUll Resolution LiveView:
Code: [Select]
0xc0f06008 : 0x4f504f5
0xc0f06014 : 0xc9c
0xc0f06084 : 0x1004b (from default 1004a)
0xc0f06088 : 0xc9d09bb
They were retrived first time taking FRSP and watch previus values of regs in adtg_gui and then after taking a regular picture while not in LV.
Timing values and resolution appears more higher than needed, is this normal?

What is the relation between Timer A/B regarding FPS and rolling shutter?
I read somewhere on forum one time and can't find anymore (Also still need to finish read half of this thread)

Here is a spreadsheet (https://docs.google.com/spreadsheets/d/1pXZGJov8V-5K-ESiRTn5Z7zgydLhvNhUMqLdLMpEKVQ/edit?usp=sharing) where I'm trying to summarize what tested and trying not to forget their outcomes
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 04, 2019, 12:51:35 AM
Great Job !!
That expected , did you try and rec a .mlv file ? Looks like a image dump
more then likely the pink lower bar with not be in the raw video
It was the same for the 5D2 , I had the same thing on full width (5632) & 4k (4096)
https://www.magiclantern.fm/forum/index.php?topic=19336.msg207777#msg207777


CMOS[2] should center it , check photo mode and see what it is.
On the 5d2 CMOS[2]0xe centers for full width (5632) and in
default 3xcrop_mode CMOS[2]0x10e centers it (2144) but in 4096(4k)
CMOS[2]0x6e centers it , I check to see by refreshing liveview with the half shutter button 
Seems the image is always right justified , and it can only be moved to the left from what I seen on 5d2 .

That all for now , I'll answer the rest of your questions later tonight
in the middle of some 4k line skipping experiments in 5x zoom (3xcrop_mode) . :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on January 05, 2019, 10:31:08 AM
Hi Reddeercity,

The squeezed mode is interesting. I'm thinking about use it in fullhd+ like (1920 x 360 up to 2400 x 432) at a high frame rate like 72 to optimise aliasing.
With an offset of 1 line in height every 1/72 second, we can reconstruct with dual-iso a good fullhd+ (1920x1080 up to 2400x1296 fps24 ) ?
I know we need to adapt mlv_dump or mlv_app to take in account the height shift but do you think it's doable in crop.c ?
Here is a schema to explain my thoughts :

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FvVNqCfp%2Fschema-ml.jpg&hash=1a0b7be655fbb9b10f33c22ada2d7f1c) (https://ibb.co/vVNqCfp)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on January 05, 2019, 10:35:34 AM
Height shifts and metadata area already under control through metadata and taken care of in Mlv App. You reach the metadata parts in crop_rec.c in here:
Code: [Select]
static unsigned int raw_info_update_cbr(unsigned int unused)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on January 05, 2019, 10:59:34 AM
Whaou , thanks Danne for the quick and precise answer !
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: a1ex on January 05, 2019, 11:11:53 AM
Maybe I'm missing the point, but I don't see any connection between the previous 3 replies. Metadata has nothing to do with benoit's diagram; maybe Danne's reply was meant for some other post?

One has to control the phase of the line skipping sequence (i.e. from a 3-line group, which line gets captured and which two lines get skipped). This is not under our control at the time of writing. If it is controllable by software, it would be from one of the ADTG registers.

To find it, one requires a way to change ADTG registers without moving the camera (such as, some sort of USB remote interface, or some register brute-forcing script running unattended). Why? You would have to identify an image shift of exactly 1/3 pixels vertically (after the line skipping).



In the mean time, the easiest way to use all physical pixels on the sensor (including 5D2) is to read every single line, and bin every 3 columns (https://www.magiclantern.fm/forum/index.php?topic=16516.msg209999#msg209999) (that is, full height 1x3 readout).

Simulation to show the difference between 3x1 (with line skipping) and 1x3 (with column binning):
binning-modes.html (https://a1ex.magiclantern.fm/bleeding-edge/lv-binning/binning-modes.html) (18.5 MB)

However, recovering the full resolution from a 1x3 readout is not exactly straightforward.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on January 05, 2019, 12:13:17 PM
Yes, 1x3 will be the way so I assume 5D mark III and maybe 700D would be the better start off cams to work on as they can can expand height with less restrictions than eosm and 100D for instance.
My metadata reply was simply referring to info passed into mlv through crop_rec.c which Mlv app follows.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on January 05, 2019, 04:12:19 PM
CMOS[2] should center it , check photo mode and see what it is.
I check to see by refreshing liveview with the half shutter button 
Seems the image is always right justified , and it can only be moved to the left from what I seen on 5d2 .
Found CMOS[4] is responsible for horizontal position on 50D, that's why i wasn't noticing any change in value while checking cmos[2]

Strange thing: after poking around cmos registers set to found what said above, for uknow reason now I get automatic centered image while entering liveview and setting full width:
Code: [Select]
Previuous automatic value:
CMOS[4] = 484
Centered Full Wide:
CMOS[4] = 404
Another Value fond early while logging photo mode, but not sure about
CMOS[4] = 40c

That expected , did you try and rec a .mlv file ? Looks like a image dump
more then likely the pink lower bar with not be in the raw video
It was the same for the 5D2
First attempt result in a standard 2000x1079 mlv video....
Discovered that to increase image size while recording, I have to manually increase resolution in movie Tab -> raw video

Max available is 3520 which give a almost good video of 3520x1078 (a couple of vertical dark lines as wrote defore) from a dump of 4768x1078

Is this how is suppose to do or I'm missing something?
@reddeercity Mlv_lite and crop_rec doesn't compile for 50D in Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018 (http://Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 06, 2019, 12:37:16 AM
Hi Reddeercity,
The squeezed mode is interesting. I'm thinking about use it in fullhd+ like (1920 x 360 up to 2400 x 432) at a high frame rate like 72 to optimise aliasing.
With an offset of 1 line in height every 1/72 second, we can reconstruct with dual-iso a good fullhd+ (1920x1080 up to 2400x1296 fps24 ) ?
I know we need to adapt mlv_dump or mlv_app to take in account the height shift but do you think it's doable in crop.c ?
Here is a schema to explain my thoughts :
Interesting idea ,
but 1280x434 is the max resolution I can go at Hi-Frame rate (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209935#msg209935) e.g. 72 (FYI - max frame rate to date is 82.5fps @ 1280x722 un-squeezed)

This the Hi-Frame rate limitations squeezed :
1888x704 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209658#msg209658) @ 48fps
1600x500 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg208955#msg208955) @ 62.5fps
1280x434 @ 82.5fps


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 06, 2019, 01:56:59 AM
Found CMOS[4] is responsible for horizontal position on 50D ......
Strange thing: .... now I get automatic centered image while entering liveview and setting full width:
Code: [Select]
Previuous automatic value:
CMOS[4] = 484
Centered Full Wide:
CMOS[4] = 404
Another Value fond early while logging photo mode, but not sure about
CMOS[4] = 40c
Great , good to know it may help other d4 cams
First attempt .... mlv video....
Discovered that to increase image size ...... increase resolution in movie Tab -> raw video
Max available is 3520 which give a almost good video of 3520x1078 (a couple of vertical dark lines as wrote defore) from a dump of 4768x1078
Is this how is suppose to do or I'm missing something?
No,  I forgot to add the extended resolutions to the source (3008, 3840,4096 & 5632) -- sorry
Add that to mlv_rec.c @ line 136 e.g. like this
Code: [Select]
static uint32_t resolution_presets_x[] = {  640,  960,  1280,  1600,  1920,  2240,  2560,  2880, 3008,  3200,  3520, 3840, 4096, 5632 };
#define  RESOLUTION_CHOICES_X CHOICES(     "640","960","1280","1600","1920","2240","2560","2880","3008","3200","3520",3840","4096","5632")
Even though your 50d can't  do 5632 it will go to the  next max. available size to you
@reddeercity Mlv_lite and crop_rec doesn't compile for 50D in Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018.zip)
I know mlv_lite didn't but wasn't sure about crop_rec  , I meaning to look in to it .
I can't even remember what error there where for mlv_lite , I need sound & HDMI support so I just kept with mlv_rec .
mlv_lite doesn't work very well with HDMI and there no sound surport in the 4K branch yet for 5D2 in mlv_lite
but I guess for the 50d there's no sound recording so you my have too look in to it your self , I my not get to it for a long time.

It took waza57 some time to get the crop_rec to work with the 5d2 , I would need to see the errors to understand
why it's not compiling .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 06, 2019, 02:09:03 AM
1888x704 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209658#msg209658) @ 48fps
Very nice!!! (edit: before I somehow missed that you got that resolution despite looking at the images 🤦)

Is it possible to disable the squeeze and have that same resolution without it?

Sorry if my spam is getting annoying, but is it possible to get higher vertical than the 2880x1080 preset if you lower the fps? Something like 2880x1200 at 21fps
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 06, 2019, 06:28:10 AM
Is it possible to disable the squeeze and have that same resolution without it?
No, at 48fps the max vertical is 704 , maybe 750
The nice think about the 5D2  with 3x5 , there no more aliasing and moire then you see in 3x3
plus at 48fps the frames are cleaner I think , good enough for smooth slow-mo @ 24fps
......  but is it possible to get higher vertical than the 2880x1080 preset if you lower the fps? Something like 2880x1200 at 21fps
In crop_mode I'm planning on 3008x1200 @ 24fps , it's almost there got sidetrack by the 4k vertical compressed height dev.
currently I have 2712x1200 @ 23.976fps & 2560x1200 @ 23.976fps (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209086#msg209086)
I'm also thinking of adding some square presets , 2144x1330 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207515#msg207515) , 2144x1586 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207553#msg207553) & 2144x1842(maybe) currently there are at 9.9fps but hope to get 24
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 06, 2019, 09:08:03 AM
I got (compressed vertical) 4416x1608 @ 23.98 working (A.R. 2.74:1) , would this resolution be useful ?
Or I can go to 4250x1722 @ 23.976 (A.R. 2.46:1) , I'm still keeping 3840x1872 (A.R. 2.05:1) & 4096x1776 (A.R. 2.30:1) @ 24fps
I just don't want too many presets or I can have both . **Note these are un-compressed vertical resolution**
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on January 06, 2019, 09:13:32 AM
No 2.35:1 preset?
By the way. Are you getting any real-time preview on any of your squeezed presets or are all of them 'framing' preview based?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 06, 2019, 05:19:29 PM
No, at 48fps the max vertical is 704 , maybe 750

I didn't mean unsqueezed equivalent of 1880x704, I meant just 1880x704 but not squeezed, so basically a wide aspect ratio. 1880x704 matches pretty well to 2.67 if it were 3x3.

Would that be possible?

The nice think about the 5D2  with 3x5 , there no more aliasing and moire then you see in 3x3

There's got to be some difference, so I would use the 3x3 just to be safe as I like wide aspect ratios anyway.


I'm planning on 3008x1200 @ 24fps , it's almost there got sidetrack by the 4k vertical compressed height dev.
currently I have 2712x1200 @ 23.976fps & 2560x1200 @ 23.976fps (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209086#msg209086)
I'm also thinking of adding some square presets , 2144x1330 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207515#msg207515) , 2144x1586 (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207553#msg207553) & 2144x1842(maybe) currently there are at 9.9fps but hope to get 24

This is good news! I tried changing fps override timers in the 2880x1080 build in first post, also the registers crop_rec lets you change, but it seems locked to height 1080. Guess I'd need to use whatever branch you are using to actually come up with these values. The name of it is not coming to my head right now.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on January 06, 2019, 09:44:43 PM
I got (compressed vertical) 4416x1608 @ 23.98 working (A.R. 2.74:1) , would this resolution be useful ?
Or I can go to 4250x1722 @ 23.976 (A.R. 2.46:1) , I'm still keeping 3840x1872 (A.R. 2.05:1) & 4096x1776 (A.R. 2.30:1) @ 24fps
I just don't want too many presets or I can have both . **Note these are un-compressed vertical resolution**
That all sounds so promising! The release date for all this will be a perfect day! 8)
If I think about what I use most, then this should be A.R.s between 16:9 and 2.35:1. But maybe 2.74:1 is useful for someone out there...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on January 08, 2019, 11:58:20 PM
let's send from each person one good of the place where he lives to Reddeercity? =) hehe.. That's absolutely incredible.

I absolutely agree with Masc about 16:9 and 2.35:1
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on January 09, 2019, 01:34:25 AM
I'm trying to get full heigth from vanilla 5x just like a1ex (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205622#msg205622) and reddeercity already did, but doesen't appears to extend image stright off like previous full width experiment.

I'm getting that pink bottom like this (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207127#msg207127) and can't get clean image like this (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207553#msg207553) by just increasing Timer B and Head timers.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 09, 2019, 04:33:30 AM
I'm trying to get full heigth from vanilla 5x just like a1ex (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205622#msg205622) and reddeercity already did,
but doesen't appears to extend image stright off like previous full width experiment.

I'm getting that pink bottom like this (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207127#msg207127) and can't get clean image like this (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207553#msg207553) by just increasing Timer B and Head timers.

That's a CMOS[1] adjustment (Vertical Offset) & Head Timers c0f0713c & c0f07150 (vertical window size for liveview)
not sure if these are the same in 50d . I get this for time to time also but sometimes refreshing liveview fixes it (half shutter)
Good that you posted , I'm currently working on those square presets for 24p (2144x1586)
Got it work at 2144x1448 @ 23.976 ( I wanted some tall 4/3 preset - I like that look)
So I took that experiment (https://www.magiclantern.fm/forum/index.php?topic=19336.msg207553#msg207553)  and built around it , getting these extended height can be difficult .

Here a procedure I use to get extended height in 5x zoom(2144x1586) on 5d2 (default 2144x1074)
1-Slow Timer'B" down to 9.98 fps (C0F06014: 0xEDC)
2-Adjust Raw Height C0F06088: 0x66A04BA (1586x2144) we read it as 2144x1586
3-Adjust the Vertical Liveview window (head timers) C0F0713C from 0x476 -> 0x676  & C0F07150 from 0x49C -> 0x69C
Need to do so math for the C0F07150 head timers e.g. 0x49c=1180 vertical default for 5d2 (1074) we need 1586+52(OB offset)=1638
0x69c=1692 , C0F0713C 0x476=1142 (default) need 1586+52(OB offset)=1638  0x676=1654 more then we need but I think you get I the idea .
4-CMOS[1] Vertical Offset adjustment , 0xE6A (default)  change to 0xBOD (just need to play around cmos[1] trial & error)
5-Increase frame rate , c0f06014 0xEDC to 0x4something etc...

That's the basics , adjust slowly -- if you flip thought the adjustments too fast you can either lockup the cam or loose the image (liveview turns black)
At best most or this is just trial & error plus a few lucky breaks  :D
 
Edit: Are you using the 50d 10-12bit build from the experimental download page
or did you use a build compiled from my source file with the add parts (redirect buffer for extended res.) for 50d ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 09, 2019, 07:12:24 AM
Still adjust different presets , trying to get more height in the 2k range
Figured out 2400x1330 @ 23.976fps real time color preview (same as canon preview) while recording raw video
record times are short , about 20 seconds at the moment (I haven't starting on Over Clocking the CF card Bus yet that's next)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F3Mq5zKz%2FVRAM28-crop-small.png&hash=3a2b9f0869a0820976116dc0ff78c0f3) (https://imgbb.com/)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F54W99qD%2FVRAM29-small-2400x1330-24fps.png&hash=50170171b138f4e43c1ecb0bf0878f68) (https://imgbb.com/)

You may wonder what the last three reg's are for , they are for the liveview preview while recording instead of a frozen liveview
c0f08184 = preview height,  I set this to the raw height +52 (the OB area)  0x467(1127)=>0x56a(1386)
c0f08188 = preview width , I set this to the total raw width+(OB Area 160) 0x907(2311)=>0xa07(2567)
c0f08518 = both preview height & width 0x4670907->0x56a0a07 ( 1127x2311)->(1386x2567) and of course we read this as 2567x1386    ;)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on January 09, 2019, 11:01:50 AM
Figured out 2400x1330 @ 23.976fps real time color preview (same as canon preview) while recording raw video

Great Reddeercity, "color realtime preview" I can't believe it ! the wimps will be glad when you release it !
Can't wait the date !  ;D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on January 09, 2019, 06:00:11 PM
That's a CMOS[1] adjustment (Vertical Offset) & Head Timers c0f0713c & c0f07150 (vertical window size for liveview)
not sure if these are the same in 50d
Not quite sure abut CMOS[1] on 50D... his default value is 0x1 and doesn't seems to change from photo mode.
Tryed to increase by little steps but no feedback from LV image.

Instead CMOS[5] looks more interesting, because it's default value is 0x628 after switching to 5x mode and increasing to 0x629 will move image up, decreasing will move image down till image get duplicated/distorted.

Unfortunately small modifications from that value doesn't cover pink are with valid data, just transalate sensor image in upper area untill it gets white...
Didn't tryed increasing/decreasing by larger value, but if this is what you get from CMOS[1] on 5d2, maybe is what I'm looking for.

Here a procedure I use to get extended height in 5x zoom(2144x1586) on 5d2 (default 2144x1074)
1-Slow Timer'B" down to 9.98 fps (C0F06014: 0xEDC)
2-Adjust Raw Height C0F06088: 0x66A04BA (1586x2144) we read it as 2144x1586
3-Adjust the Vertical Liveview window (head timers) C0F0713C from 0x476 -> 0x676  & C0F07150 from 0x49C -> 0x69C
Need to do so math for the C0F07150 head timers e.g. 0x49c=1180 vertical default for 5d2 (1074) we need 1586+52(OB offset)=1638
0x69c=1692 , C0F0713C 0x476=1142 (default) need 1586+52(OB offset)=1638  0x676=1654 more then we need but I think you get I the idea .
4-CMOS[1] Vertical Offset adjustment , 0xE6A (default)  change to 0xBOD (just need to play around cmos[1] trial & error)
5-Increase frame rate , c0f06014 0xEDC to 0x4something etc...
Yes, basically it's what I'm doing, except for CMOS. From photo mode 0xc0f06088 = c9d09bb, 0xc0f06014 = c9c and 0xc0f06008 = 4f504f5

Noticed that timers appears to have a little overhead based from math (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209780#msg209780) you pointed to me before, should I try to reduce timer and resolution (to mach cr2 and reduce black area) later when I get Full Resolution working?

That's the basics , adjust slowly -- if you flip thought the adjustments too fast you can either lockup the cam or loose the image (liveview turns black)
At best most or this is just trial & error plus a few lucky breaks  :D
Cycling through video mode will usually help to get LV working when gets completely dark.

Edit: Are you using the 50d 10-12bit build from the experimental download page
or did you use a build compiled from my source file with the add parts (redirect buffer for extended res.) for 50d ?
I'm using your source files with the addittion of 50D's stubs (https://bitbucket.org/hudson/magic-lantern/commits/db4ee396f4a261d33688b528c39447682f871a07?at=raw_video_10bit_12bit_LVState) for CONFIG_EDMAC_RAW_PATCH support.
I still need to recompile with what you told me before (https://www.magiclantern.fm/forum/index.php?topic=19336.msg210179#msg210179) to add extendeds resolutions.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 09, 2019, 07:33:26 PM
@reddeercity are you using the iso-research branch with adtg gui module to come up with these settings? I'm wanting to try... I see you posting screenshots of your settings which seem like a good starting point.

Also have you been able to create any presets (that can be selected in crop_rec menu) since that time Danne tried to put it in crop_rec.c?

I have been reading the code and competely confused how a preset is added to the menu. Why can't it just be a struct with some values...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 10, 2019, 06:21:16 AM
@Ilia3101 I'm working on the code , lots of parts to deal with .
if you what to try , use the build on the first page of this thread then
download the ADTG_GUI.mo from https://builds.magiclantern.fm/modules.html
that's the one I use . Only load mlv_rec.mo & adtg_gui.mo -- Do Not load the crop_rec.mo

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 10, 2019, 06:54:30 AM
Can somebody check this please , I'm not sure about the "case cmos_new[1]:"
not sure if I need to add it here or somewhere else
I'm using waza57 crop_rec source 5d2_crop_rec.c (https://bitbucket.org/waza57/magic-lantern/src/8bfbb0ca228e11f091869df09a1b8f16dd36b508/modules/crop_rec/crop_rec.c?fileviewer=file-view-default)
Starting at Line 1508 of crop_rec.c (https://bitbucket.org/waza57/magic-lantern/src/8bfbb0ca228e11f091869df09a1b8f16dd36b508/modules/crop_rec/crop_rec.c?fileviewer=file-view-default#crop_rec.c-1508) is where I made the changes , I'm using the values from this post here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg210327#msg210327)
Code: [Select]
static inline uint32_t reg_override_zoom_fps(uint32_t reg, uint32_t old_val)
{
    /* attempt to reconfigure the x5 zoom at the FPS selected in Canon menu */
    if (is_5D2)
    {
     int timerA = 679 ;         //waza57 good values for 23.976: 804- 1245  and 807 - 1237
     int timerB = 1471 ;     
     int a = reg_override_fps_nocheck(reg, timerA, timerB, old_val);
        if (a) return a;
           switch (reg)
            {
                /* raw resolution (end line/column) */
                /* X: (3072+140)/8 + 0x17, adjusted for 3072 in raw_rec */
               
                case cmos_new[1]:
                    return (old_val & 0x0000000) + 0xb0d  ;

                case 0xC0F06084:
                    return (old_val & 0x0000000)  ;
               
                case 0xC0F06088:
                    //return (old_val & 0xFFFF0000) + 0x630 ;
                    return (old_val & 0x00000000) + 0x56a053a;   //0x4b00648 or origin value chang nothing

                 /* HEAD3 timer */
                case 0xC0F0713c:
                    return 0x566 + YRES_DELTA + delta_head3;

                /* HEAD4 timer */
                case 0xC0F07150:
                    return 0x58c + YRES_DELTA + delta_head4;

                case 0xC0F08184:
                    return (old_val & 0x0000000) + 0x56a ;

                case 0xC0F08188:
                    return (old_val & 0x0000000) + 0xa07 ;

                case 0xC0F08518:
                     return (old_val & 0x0000000) + 0x56a0a07 ;                   
            }

Here Line-594 (https://bitbucket.org/waza57/magic-lantern/src/8bfbb0ca228e11f091869df09a1b8f16dd36b508/modules/crop_rec/crop_rec.c?fileviewer=file-view-default#crop_rec.c-594) there's a cmos[1] change , I thinking that's where I should put the new cmos[1] value
Code: [Select]
  if (is_5D2)
                {
                cmos_new[1] = PACK12(42,57);//0xB0D;    /* pink highlights without this */
                cmos_new[2] = 0x10E;    /* read every column, centered crop */     //waza57 cmos 6
                break;


Edit: I updated my source with the 50d stubs for "CONFIG_EDMAC_RAW_PATCH" support
Crop_Rec-4k-5d2-50d-magic-lantern_1-9-2019.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-50d-magic-lantern_1-9-2019.zip)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: theBilalFakhouri on January 10, 2019, 10:52:00 AM
Figured out 2400x1330 @ 23.976fps real time color preview (same as canon preview) while recording raw video
You may wonder what the last three reg's are for , they are for the liveview preview while recording instead of a frozen liveview
c0f08184 = preview height,  I set this to the raw height +52 (the OB area)  0x467(1127)=>0x56a(1386)
c0f08188 = preview width , I set this to the total raw width+(OB Area 160) 0x907(2311)=>0xa07(2567)
c0f08518 = both preview height & width 0x4670907->0x56a0a07 ( 1127x2311)->(1386x2567) and of course we read this as 2567x1386    ;)

This is great! Do you mean also the real-time preview now is uncropped ? It shows the entire frame in real-time?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 11, 2019, 04:08:45 AM
This is great! Do you mean also the real-time preview now is uncropped ?
Yes , until recording raw , then just like the default crop_mode it center crops
but I never tried with the overlays enabled as I have "kill overlays on recording"
in mlv_rec , get me a extra 25% write speed .
So it may with overlays  :) I'll check next time I have it loaded up.
Here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg209086#msg209086) I posted a iPhone video or the lcd screen showing the Liveview as I record 2400x1200 @ 23.976fps .

I know before with default crop_mode (2144x1074) if you press half shutter , you get a color preview with correct full framing
while recording , so I'm not sure it there's ML code that does the resizing or if it's a reg , maybe in the "preview regs"
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on January 11, 2019, 06:14:58 AM
Still adjust different presets , trying to get more height in the 2k range
Figured out 2400x1330 @ 23.976fps real time color preview (same as canon preview) while recording raw video
record times are short , about 20 seconds at the moment (I haven't starting on Over Clocking the CF card Bus yet that's next)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F3Mq5zKz%2FVRAM28-crop-small.png&hash=3a2b9f0869a0820976116dc0ff78c0f3) (https://imgbb.com/)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F54W99qD%2FVRAM29-small-2400x1330-24fps.png&hash=50170171b138f4e43c1ecb0bf0878f68) (https://imgbb.com/)

You may wonder what the last three reg's are for , they are for the liveview preview while recording instead of a frozen liveview
c0f08184 = preview height,  I set this to the raw height +52 (the OB area)  0x467(1127)=>0x56a(1386)
c0f08188 = preview width , I set this to the total raw width+(OB Area 160) 0x907(2311)=>0xa07(2567)
c0f08518 = both preview height & width 0x4670907->0x56a0a07 ( 1127x2311)->(1386x2567) and of course we read this as 2567x1386    ;)

Are these the complete set of registers changed? Could you recheck if any setting is missed when selecting overridden regs only?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 14, 2019, 10:58:13 PM
@reddeercity Is it techninally possible to get slow motion 48+ fps in crop mode? If I were to copy your slow motion registers while in 5x would it be effective? Or does that need something completely different


@a1ex We have been discussing with Danne about making presets for crop_rec out of the sets of registers that reddeercity has been finding. It seems really difficult, can there not be a way to just have them in some c structs/arrays?

soething like
Code: [Select]
preset_struct presets[ ] =
{
    {
        .name = "preset1",
        .num_registers = 2,
        .registers = {register1, value1, register2, value2}
    },
    {
         .name = "preset2",
...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 15, 2019, 06:35:22 AM
@reddeercity Is it techninally possible to get slow motion 48+ fps in crop mode? If I were to copy your slow motion registers while in 5x would it be effective?
Or does that need something completely different
Yes needs something different, 48fps is dependent on line skipping & vertical compressed (3x5) which only work will in 3x3
you can force line skipping  in 3x crop but can't produce a useable image (being trying for 2 weeks now)
The best to hope for in crop mode would be a very small crop window e.g. 1280x430 (crop factor of about 4.5 which I have no interest in)
The simple fact that the sensor is very slow (96MP/s) kept it from any meaning full resolutions & or hi-frame rate .
This is a fact that can't be changed at present , Overclocking the sensor frequency would be the only way (24->32 maybe ?)
a1ex would the only one that would know for sure.
But you got to ask yourself , is worth it ? there would need to be more voltage among other variable (just like overcocking PC CPU)
Can it handle it or will it burn up ?

But you can certainly explore the possible , All my note are here on my thread .



..... discussing with Danne about making presets for crop_rec out of the sets of registers that reddeercity has been finding.
It seems really difficult, .....
No it not difficult , I got it figured out just been too busy to finish coding it .
D4 cams work a little different then d5 cams in crop_rec . (this is no small thing , it took @waza57 some time to get it to work , more then a mirror hack)
beside that , I not very impressed with 4k compressed vertical (4096x590)
It seems to produce more artifact then standard 3x3 (1856x1248)  , I guess because of the different pixel binding then d5 -- it's worst then d5 cams .
So I guess just because you can implement something doesn't mean you should .  :-\
I'm looking at this from a video production side (that why I won't have anything that doesn't record @ 24fps - besides Full Res (5632x3755@4.5fps)
I'm re-evaluating all my presets , sorry for the delay but life come first  :)
As this part of the development is coming to a close for the 5D2 , other d4 cams with users need to catch up on there own (I'll still help where I can).

After I finish the crop_rec for the 5d2 , I'll spend my time now on overclocking CF card bus & Lossless compression .
I know the limitations of the sensor now  , I pushed it as far as it can go (I can see a cam upgrade for me in the future maybe 5d3 or 5d4)


Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 16, 2019, 12:25:47 AM
Yes needs something different, 48fps is dependent on line skipping & vertical compressed (3x5) which only work will in 3x3
you can force line skipping  in 3x crop but can't produce a useable image (being trying for 2 weeks now)
The best to hope for in crop mode would be a very small crop window e.g. 1280x430 (crop factor of about 4.5 which I have no interest in)
The simple fact that the sensor is very slow (96MP/s) kept it from any meaning full resolutions & or hi-frame rate .

Assuming 96 is a reachable limit, here's what we would be able to get at 60fps:

Aspect         Max resolution
2.67:12064x774
2.35:11936x824
16:91680x945

This sounds really appealing if possible to me. I'm sure it's probably not possible to reach that limit so perfectly, but those resolutions might be doable at something lower like 50fps.

But you can certainly explore the possible , All my note are here on my thread .

What you have been doing is definitely very helpful.


No it not difficult , I got it figured out just been too busy to finish coding it .
D4 cams work a little different then d5 cams in crop_rec . (this is no small thing , it took @waza57 some time to get it to work , more then a mirror hack)

:(

It is difficult to understand though, I still have no clue how to make presets.

I'm looking at this from a video production side (that why I won't have anything that doesn't record @ 24fps - besides Full Res (5632x3755@4.5fps)

I would personally like to make lower fps ones just for higher, but not full, resolution recording. It works if there's no speech or fast movement.

I'm re-evaluating all my presets , sorry for the delay but life come first  :)

Understandable :)

After I finish the crop_rec for the 5d2 , I'll spend my time now on overclocking CF card bus & Lossless compression .

Any of those two things would be AMAZING 😍

I know the limitations of the sensor now  , I pushed it as far as it can go (I can see a cam upgrade for me in the future maybe 5d3 or 5d4)

You definitely deserve the upgrade.

I am starting not to want to upgrade ever though, 5D mark 2 is too much fun. How endlessly can the 11 year old camera be pushed?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on January 16, 2019, 12:55:47 PM
I pushed it as far as it can go (I can see a cam upgrade for me in the future maybe 5d3 or 5d4)

get the 5D4!!!  let's not wait until it is 11 years old before it is supported!
you're the man to make it happen!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ItsMeLenny on January 17, 2019, 02:16:20 AM
96 isn't possible on any camera https://docs.google.com/spreadsheets/d/16cgnRivbUv7nA9PUlCLmLdir3gXdIN3pqzCNAAybepc/edit#gid=5
in digic IV. Unless anybody's tried overclocking the SD card speed.
Probably wonldn't need to if RAW compressed (or lossless whatever stupid name people were calling it) could work.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 17, 2019, 06:36:05 PM
96 isn't possible on any camera https://docs.google.com/spreadsheets/d/16cgnRivbUv7nA9PUlCLmLdir3gXdIN3pqzCNAAybepc/edit#gid=5
in digic IV. Unless anybody's tried overclocking the SD card speed.
Probably wonldn't need to if RAW compressed (or lossless whatever stupid name people were calling it) could work.

This isn't MB/s card speed, it's sensor speed- Megapixels per second. I was just coming up with some theoretically possible resolutions/framerates that I want to try and get on the 5D2 in crop mode. They wouldn't record continuously of course, about 6 seconds in 10 bit (but it is 60fps, so a 14 second long shot once slowed down to 24)

There's also a chance that none of this is possible at all in crop mode.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 19, 2019, 06:45:39 AM
I'm looking at keeping 4k (4096x590=>4096x1770 23.976fps) or as I call it
Vertical compressed , don't like the name 1x3 or 3x1 or what ever it's called (less confusing)
I did a short 4 min talking head clip (me talking about 4k on the 5d2  :) ) Yea that continuous @69MB/s write speed @ 10bit uncompressed
I found it to be not as bad as I thought , yes there a very small amount of aliasing
but with some heavy grading and anti-aliasing filters is more then useable
thou you must select your shots wisely to avoid this little issue .(fine patterns etc. .. you know the drill)
Here a frame of the close to final grade , still needs a little tweaking .

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FV9DwCbZ%2F5-D-Mark-ii-Real-4-K-small.png&hash=49c1e8260df4014315509f101343d87a)
5D Mark ii Real 4K.tiff (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5D%20Mark%20ii%20Real%204K.tiff)

Workflow:
MLV App 1.4 wins , to exact dngs to full height (590->1770)
Imported the dng's in to Adobe CS6 A.E. ACR set project to 32bit float
(I tried mlv app first , but I can't get the image to look right the skin tone where really bad & had had too much noise in the shadow ,
sadly I when back to A.E. so slow 6Hrs to generate 6300 4k tiff's  :( compared to 90 mins on mlv app maybe it better on the MAC never tried).
Imported the tiffs in to Apple compressor , made a real ProRes4444XQ (Apple ProRes is much better , 16bit+alpha)
The real ProRes4444XQ can be real pushed hard for grading , take a lot to break the image .
Imported in to FCPX edit & final grade , added my LUT there , "ArrilInterneg_KodakVision_sRGB.cube .

So do the other 5D2 users think I should keep 4K preset in compressed vertical ?
The sample above should be noted that I was a little out of focus (hard to do by yourself  :D )

FYI: 4mins of 4096x590@ 23.976fps = 17.7GB (mlv_rec file) that got blown up to 350GB of tiffs = 40GB ProRes4444XQ
So at 4096x1770 , 1 dng = 9MB , 1 Tiff = 58MB x6300 frames
Cam settings 400 ISO 1/50th 180-degree shutter , Canon Ef24-70 f2.8L @ 24mm f4.0
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on January 19, 2019, 01:30:04 PM
So do the other 5D2 users think I should keep 4K preset in compressed vertical ?
Any of your posted presets would be nice to try out for me. But unfortunately I don't get one single preset to work over here...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 19, 2019, 02:16:26 PM
So do the other 5D2 users think I should keep 4K preset in compressed vertical ?

First impression when I opened it: wow this looks sharp, second impression: Oh there's a lot of alising (even on the out of focus monitor).

I still think the 1:1 presets look better, though you could keep this one for a wider field of view. Also a softer lens might reduce aliasing.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: benoit on January 19, 2019, 02:58:16 PM
As presets need to be frequently updated, I like the Ilia idea of creating a struct to store different presets.
If it's possible to do it, it's more suitable, it's open, it's simple (no need to compile every time).
Indeed, we can exchange an xml file (or whatever file format)  that store 5D2 presets between users.
IMHO, I think 3K/UHD raw developpement and registers discover will be more easy and quickly spread over users.

David
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 20, 2019, 01:59:50 AM
As presets need to be frequently updated, I like the Ilia idea of creating a struct to store different presets.
If it's possible to do it, it's more suitable, it's open, it's simple (no need to compile every time).
Indeed, we can exchange an xml file (or whatever file format)  that store 5D2 presets between users.
IMHO, I think 3K/UHD raw developpement and registers discover will be more easy and quickly spread over users.

David

Well with my idea (https://www.magiclantern.fm/forum/index.php?topic=19336.msg210543#msg210543) (C structs in the source code) you'd still need to compile it each time, but it would still be massively easier than what it is now (loads of random bits spread all over the code where you need to add stuff).

And your idea of having it in a file is amazing if it could come true, but it seems unlikely to me that anyone able to do that will find time (and for my idea). It also requires a format simpler than xml, maybe csv, or even a binary format that would need to be edited with a hex editor.

I tried recreating reddeercity presets with adtg_gui (crop rec disabled), no success yet, can't find some of the registers 🙁
@reddeercity Also are the adtg_gui screeenshots showing all the registers you changed?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 20, 2019, 06:26:50 AM
@Ilia3101 , Your right too much aliasing , check out the sample below  -- better ?
Yes when I post Reg's screen shots it all the reg's  needed for the what ever preset I was working on.
No sure why you can't reproduce , 5D2 is really touchy , if you adjust reg's out order the cam will lock up or will not respond .
You have to enable (in the advanced tab) ENGIO registers , then refresh , view or play a h264 or cr2 file to update the reg's
Rule of thumb : First thing Slow down Timer"A" first then Timer"B" , I usually do all my work around 10fps then after I set raw rez.
and head timers I increase timers "B" to the correct frame rate (23.976) (but slowly) .
Yes "C" code can be hard to understand sometimes , but there is a reason for everything  ;D
If you read the crop_rec code you will see that it  needs calls from adtg to apply the modified regs.


Ok I'm very close to getting 3xcrop (5xzoom) to work with Lineskipping 
I had to start in 1:1(3x3 FHD) set the timers , resolution & cmos[2] reg
Then switch to 3x crop_mode (5x zoom) set to adtg12[100c] to line skipping (0x5->0x6)
Adjust frame blanking adtg1[106] 0x301->0x1 lastly cmos[5]
Seem to be cleaner , haven't notice any aliasing yet but I'm still adjusting
here a sample to pixel peep

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fk2T4Cby%2FM19-2238-frame-47-small.png&hash=d359a4ee228d9d16d8614b313b97f6e7) (https://imgbb.com/)
3x_crop_mode_Line-Skipping_4096x1776_M19-2238_frame_47.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/3x_crop_mode_Line-Skipping_4096x1776_M19-2238_frame_47.png)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on January 20, 2019, 07:42:49 PM
yep!

definitely looks much better!!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 20, 2019, 08:09:44 PM
does look better
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 21, 2019, 10:50:00 AM
Yes when I post Reg's screen shots it all the reg's  needed for the what ever preset I was working on.

good

No sure why you can't reproduce , 5D2 is really touchy , if you adjust reg's out order the cam will lock up or will not respond .
You have to enable (in the advanced tab) ENGIO registers , then refresh , view or play a h264 or cr2 file to update the reg's
Rule of thumb : First thing Slow down Timer"A" first then Timer"B" , I usually do all my work around 10fps then after I set raw rez.
and head timers I increase timers "B" to the correct frame rate (23.976) (but slowly) .

Thank you for this, really useful! I had enabled ENGIO, but had no idea about the playing a file trick. Also had no idea about the order of setting them. I think this will help a lot next time I try.

Yes "C" code can be hard to understand sometimes , but there is a reason for everything  ;D
If you read the crop_rec code you will see that it  needs calls from adtg to apply the modified regs.

I still think it could be modified to make those calls based on some data in a simple to understand struct. I would like to do this if I ever get an understanding of how crop rec presets work right now.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on January 21, 2019, 03:03:34 PM
Well one way to proceed is to create actual presets in crop_rec.c. From there one can alter registers from within a sub menu structure. Then include all relevant registers and connect them to the presets so it´s possible to modify anything but from a starting point where all registrys are already set.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 22, 2019, 07:37:06 AM
Ok I got crop_rec to work with my new preset 2.4k (2400x1330 @ 23.976)
But I do have a problem , waza57 added a rom hack to increase height in 5x Zoom
before a1ex implemented "Raw_Slurp" Redirect Buffer for extended resolution in 5D2/D4 Cam .
As a result my preset is  2368x1266 @24.028fps  :(
Code: [Select]
Active Area     : 0 0 1266 2368
Frame Rate      : 24.028

So I'm currently going thought the crop_rec code line by line to figure out how to disable it/remove it
as it's no longer needed as a result the 2 hacks are fighting each other causing conflict

@waza57 is there a quick way to disable the rom hack you added ?

So once that rom hack is disable , I can implement all my preset including
4K line skipping in 5x zoom , got that to work now @ 23.976fps (4096x590=>4096x1770)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fp1ZQ7r6%2FVRAM1-small.png&hash=4f2544d0670a7807c96dbe30f03256df) (https://imgbb.com/)

We are almost thought the forest now and I can see a clearing with rolling grasslands to play in  :))
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 23, 2019, 07:20:17 AM
Boy this ROM hack from waza57 for height is really hard to disable , more I think I'm close the more error's I get  & can't compile  :(
On the Bright side  :D My preset code structure works , after I found where to add the code.

2400x1330 23.976fps +unfrozen liveview
Code: [Select]
line_592-598 
                  if (is_5D2)
                {
                cmos_new[1] = 0xB0D;    /* pink highlights without this */
                cmos_new[2] = 0x10E;    /* read every column, centered crop */     //waza57 cmos 6
                break;
                }
                break;   
   
 
Code: [Select]
****Line_685-693****
/* raw buffer centered in zoom mode */
            case CROP_PRESET_CENTER_Z:
                cmos_new[1] = (!is_5D2)
                  ? 0xB0D          /* vertical (first|last) */
                  : -1;
                cmos_new[2] = (!is_5D2)
                ? 0x10E            /* horizontal offset (mask 0xFF0) */
                : -1;
                break;
Code: [Select]
****Line_1508-1544***
static inline uint32_t reg_override_zoom_fps(uint32_t reg, uint32_t old_val)
{
    /* attempt to reconfigure the x5 zoom at the FPS selected in Canon menu */
    if (is_5D2)
    {
     int timerA = 680 ;         
     int timerB = 1472 ;     
     int a = reg_override_fps_nocheck(reg, timerA, timerB, old_val);
        if (a) return a;
           switch (reg)
            {
                /* raw resolution (end line/column) */
                /* X: (3072+140)/8 + 0x17, adjusted for 3072 in raw_rec */
               
                 case 0xC0F06084:
                    return (old_val & 0x0000000)  ;
               
                case 0xC0F06088:
                    //return (old_val & 0xFFFF0000) + 0x630 ;
                    return (old_val & 0x00000000) + 0x56A053A;   //0x4b00648 or origin value chang nothing

                case 0xC0F0713C:
                    return (old_val & 0x0000000)  + 0x566;

                case 0xC0F07150:
                    return (old_val & 0x0000000) + 0x58C;
               
                case 0xC0F08184:
                    return (old_val & 0x0000000) + 0x56A;

                case 0xC0F08188:
                    return (old_val & 0x0000000) + 0xA07;

               case 0xC0F08518:
                    return (old_val & 0x0000000) + 0x56A053A;

               }

Plus another Good thing I discovered tonight , after testing my Reg's with ADTG_gui to make sure I have all of them right
I thought I would test HDMI to my Zacuto EVF and see if it works & Yes! I can preview in real time over the HDMI
while recording raw @ 2400x1330 23.976fps  :D
Even pass it thought to my Ninja HDMI recorder , thou the image is gray scale when recorded but Half Shutter press give
a color preview . It's still center crop but at least you can have finer focus level special if you have diopter as I do for the EVF
Still haven't figured out how to scale it to LCD(720x480) or HDMI(1920x1080) yet .

I'll start fresh tomorrow on the crop_rec.c , hopefully I'll has a better luck .


Edit: I see there maybe a quick way to disable the ROM hack
edmac-memcpy.c , fio-ml.c
          I need to disable call to function "clean_d_cache()" otherwise i can't patch ROM addresses
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on January 23, 2019, 01:07:42 PM
I really hope you get a 5D4...  Your persistence and determination are badly needed for that unfortunate unappreciated BEAST!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 24, 2019, 01:08:03 AM
Ok that didn't work , there more that's needed to override this going to have to take a drive deep .
It may be more then just Crop_Rec.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 24, 2019, 08:00:01 AM
Found the ROM_Hack from waza57 , now I open up a can of worms  :(
just freezes in x5 zoom with crop_rec enable , before at least I had a image , thou it was all messed up.
going to have to back track again , this maybe over my head , I thought by disabling the hack I could get this to work.
not too sure now .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on January 24, 2019, 03:01:11 PM
Boy this ROM hack from waza57 for height is really hard to disable , more I think I'm close the more error's I get  & can't compile  :(
On the Bright side  :D My preset code structure works , after I found where to add the code.
I was wondering if isn't better to make Digic 4 camera build on top of official crop_rec_4k branch? This is something should be done before or after.

At the moment I can remember @dfort for lossless compression here (https://bitbucket.org/daniel_fort/magic-lantern/commits/branch/crop_rec_4k_Digic4) and a new one here (https://bitbucket.org/daniel_fort/magic-lantern/commits/branch/compressed_raw_digic4) , which is a fork of crop_rec_4k if I'm not wrong; then there is waza57 repository (https://bitbucket.org/waza57/magic-lantern/branch/crop_rec_4k_5D2), which I'm not of what is capable of doing and I imagine that @reddeercity is working on top of that because is the one which compile, is it right?

Can someone make me an overview of the difference on those repository and what is needed to make d4 compile on top of crop_rec_4k?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on January 24, 2019, 05:13:38 PM
Hi, Reddeercity, this is about zoom jumping out (the problem of my camera I described)
https://drive.google.com/file/d/1aa8kzCQueoRzHM1PoiVRr0jjbkSrG8Za/view?usp=sharing
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 25, 2019, 06:53:25 AM
Success !! disabled the "Rom Hack" and I can use the "raw_slurp" redirect buffer for extended rez.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on January 25, 2019, 08:30:41 AM
Adding above gives this when compiling:
Code: [Select]
           if (!patch_active)   waza57 changes to effective update for 5d2
                                       ^
crop_rec.c:1660:71: error: invalid suffix "d2" on integer constant
           if (!patch_active)   waza57 changes to effective update for 5d2
                                                                       ^
crop_rec.c: In function 'crop_rec_init':
crop_rec.c:2313:9: error: 'RAWROMHEIGHT_WRITE' undeclared (first use in this function)
         RAWROMHEIGHT_WRITE = 0xFFCAA1F4;
         ^
crop_rec.c:2314:9: error: 'MEM_RAWROMHEIGHT' undeclared (first use in this function)
         MEM_RAWROMHEIGHT = 0x00000467;
         ^
crop_rec.c:2316:9: error: 'RAWROMWIDTH_WRITE' undeclared (first use in this function)
         RAWROMWIDTH_WRITE = 0xFFCAA1EC;   // waza57 patch ROM for more height
         ^
crop_rec.c:2317:9: error: 'MEM_RAWROMWIDTH' undeclared (first use in this function)
         MEM_RAWROMWIDTH = 0x00000FCE;

Did you also disable this?
Code: [Select]
       // RAWROMHEIGHT_WRITE = 0xFFCAA1F4;
       // MEM_RAWROMHEIGHT = 0x00000467;
       
       // RAWROMWIDTH_WRITE = 0xFFCAA1EC;   // waza57 patch ROM for more height
       // MEM_RAWROMWIDTH = 0x00000FCE;

Please upload a working crop_rec.c version that you know compiles.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 29, 2019, 07:18:52 AM
Ok Finally got Crop_Rec working correctly on 5D2 with "raw_slurp" redirect buffer !
I started fresh with a simple  3k @ 23.98fps preset (3008x1074 @ 23.98fps)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FGnKz6KB%2FVRAM16-small.png&hash=b2e60c0e3e40f1b8b0e48e175c60942f) (https://imgbb.com/)

magiclantern-crop_rec-3k-5D2-eXperimental.2019Jan28.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/)
Updated file
magiclantern-crop_rec-3k-5D2-eXperimental.2019Jan31.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/)
Updated File
magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip)

Unfrozen Liveview while recording raw video  :D

I was having problem with c0f0713c & c0f07150 , as I have this all wrong in the code
and figured out that if you reduce 7150 more then 713c
you actually reduce over head for more frame FPS , that's how I squeezed out 23.98 @ 3008x1074 ,
if you leave those reg's alone the best you can do is 18fps at the same resolution .

The other big problem I have (haven't solved yet) is I need different "CMOS[1]" vertical offsets
setting then the one in "crop_rec" seems it just call's the one vertical offset in default 5x Zoom mode (0xE6A)
where in fact I need to have at least 3-4 different vertical offset (one per preset)
Unless there's something I missed , I just worked around the default CMOS[1] offset
and adjust from there . That why I haven't increase the vertical from default.

So this is the first test build for everyone to try , You only need to load
Code: [Select]
5D2_212.sym
crop_rec.mo
mlv_rec.mo

Same as before , canon menu 30fps -- enable 3.5k preset then enter in to
5x zoom , should take a few seconds before it comes up , half shutter will give a full color preview.
when you start recording , there should be a short 1-2 second delay then the preview will switch to real
time (canon preview) in b/w , as noted in my other posts it a center crop , but it is in real time  8)

Do Not ! enable "frame override" as this will lockup/freeze cam .
FYI - there's no mlv_lite.mo right now ,I need to add lossless to it before it can compile .
Test it as normal , let me know what breaks if anything . I'll add more preset after I figure out how
to code CMOS[1] offset's correctly .

@danne not sure where you when wrong , I checked the code I posted and it looks right
but I check it againist the source . when I have every thing done I'll post my code , it far too messy right now. :P
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 29, 2019, 10:46:01 AM
Incredible work @reddeercity!!! Can't wait to try it out.

3008x1074x24 is only 77Mpix/s, so there should still be plenty of  room for height increases if the overhead can be reduced more.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on January 29, 2019, 03:54:02 PM
Thank for Anything that you give us RedDeerCity but What's the best 1:1 Resolution? I do notice if I'm going to use x5 then to get the equlavate of a 50mm I need to use 24-35mm right?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on January 29, 2019, 08:11:51 PM
Thank you @reddeercity! I tried it out:
so far it is working over here. Mostly it is working with superb picture quality. In one test clip the first 2 pictures were corrupted and after some seconds maybe 10 frames came in as "corrupted bundle". Other test clips were 100% fantastic.
This might not be of interest: I can't quit the crop mode by using zoom button. I have to use menu button. But I think, to make this safe was not your goal ;) (no real problem so far)
Keep up the great work!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on January 29, 2019, 08:16:53 PM
About to test it.

@masc you are right, leaving 5x stopped working. It did work some time ago though... used to be able to get to 10x
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on January 30, 2019, 01:19:37 AM
My tests went well! I had one freeze/battery pull but it happened when I was bumping my camera around a lot getting it onto a tripod shoe so I'm not exactly sure what caused it.

I was surprised at how long I could record with my Lexar 1066x. The icon was red but it just kept going. I got to 2 minutes before eventually stopping it myself.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 30, 2019, 05:45:08 AM
@ masc & Ilia3101 thanks , yea not too worried about safety right now , you know Bleeding Edge stuff  :D
The best way to exit crop_rec is to press the play button twice (once to enter in player mode & once to exit player mode)
this will exit to 3x3 (1:1 FHD) without any cam freeze/lockup that needs a battery pull . Then you can enter into crop_rec
again by pressing the 5x zoom .


I was surprised at how long I could record with my Lexar 1066x. ........  got to 2 minutes before eventually stopping it myself.

Will that cool ! I only tested on my slow 90MB/s Sandisk extreme pro  , just check on my 32GB Lexar 1066x card .
Holy Cow ! your right , ml is only expecting around 300 frames , I stop at 10GB 3300 frames(2:19 min) 
WOW that's continuous  :o , never thought 3008x1074 @ 23.98fps would be special when 2880x1080 was only good for 20-30 seconds .

Must be because of reducing the head timers 3 & 4 which reducers the over head .
ML reports it takes 92.3MB/s @ 3008x1074 23.98 but in the setting file I see the write speed reported was 75.8MB/s , strange .

Another bonus  - HDMI works while recording raw  in crop_rec 3k (3008x1074) , check with my HDMI Zacuto EVF .
But you need to enable "Force HDMI to VGA" so that changes the HDMI out put from 1920x1080 to 720x480.
It really very surprising that HDMI works as well as it does in 3k plus recording continuous , that just blow me away :))

Edit:
For others who may have missed it , 3008x1074 build is available now , details here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg211130#msg211130)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on January 30, 2019, 01:52:39 PM
well done!!!

Again>
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: coco770108 on January 30, 2019, 04:53:21 PM
That is Incredible !!!
THX @reddeercity!!!
cannot wait to try it~~
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: stokis on January 31, 2019, 11:57:24 AM
I tested it, filmed many clips. Strange, but when i play the clips on my pc it plays in fast motion. It looks like in reality 3008x1074 is filmed in ~17 fps, but compressed in 23.976 and plays back in fast motion.
Try it yourself. Film some animal or human, you will see it will play back not in normal pace but in fast motion.

I calculated real file size vs time, and it is ~99 MB/s. But when i film, the camera display shows me ~72 MB/s.

p.s. 2880x1080 10bit worked perfect for me, with normal motion as 23.976 supposed to work.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Bender@arsch on January 31, 2019, 05:14:32 PM
I have tried the new Preset. In the beginning i must say perfekt...

But... the record timer is to slow, maybe x0.5 -> in the postproduktion the soundline is in normal speed, but the moviespeed maybe x2.  I thing there are not 23.98 FPS, but ~ 12 FPS.

sry for bad english
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on January 31, 2019, 09:00:08 PM
Thank for the report
 I'll check it out and see what's going on
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 01, 2019, 06:07:49 AM
Found the problem in the Head Timers , had c0f07150(Head timer 4) below the valve( of c0f0713c (Head Timer 3)
I relaxed Head Timer c0f07150 to the same valve a c0f0713c (0x467) now everything is running at the right speed .

Here the updated build , 3008x1080 @ 23.98 fps
magiclantern-crop_rec-3k-5D2-eXperimental.2019Jan31.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/)
Updated File
magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip)
Updated , see  post #512  (https://www.magiclantern.fm/forum/index.php?topic=19336.msg211379#msg211379)

Ran a few tests , looks ok but no more continuous write speed , sorry  :-[
that was side effect of the messed up head timers .
I'm getting around 300-400 frames right now , until I get on with cf card over clocking & lossless compression .

Everything should be the same as before with real time preview while recording raw @ 3k 24p
HDMI should also work , with hdmi forced to VGA (720x480) .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on February 01, 2019, 05:46:17 PM
Just stumbled onto something that might help Digic IV.

src/propvalues.c
Code: [Select]
        //#ifdef CONFIG_DIGIC_IV - fixme
        return version[0] == '4';
        //#endif

If you look at the code for any Digic V camera you'll find this in the platform/[camera]/internals.h file:

Code: [Select]
/** This camera has a DIGIC V chip */
#define CONFIG_DIGIC_V

This is not in the internals.h for Digic IV cameras. Code like this could be simplified:

src/raw.c
Code: [Select]
#if defined(CONFIG_5D2) || defined(CONFIG_50D) || defined(CONFIG_60D) || defined(CONFIG_550D) || defined(CONFIG_500D) || defined(CONFIG_600D) || defined(CONFIG_1100D) || defined(CONFIG_7D)
#define RAW_PHOTO_EDMAC 0xc0f04208
#endif

#if defined(CONFIG_5D3) || defined(CONFIG_700D) || defined(CONFIG_6D) || defined(CONFIG_EOSM2) || defined(CONFIG_EOSM) || defined(CONFIG_650D) || defined(CONFIG_70D) || defined(CONFIG_100D)
#define RAW_PHOTO_EDMAC 0xc0f04008
#endif

To this:

Code: [Select]
#ifdef CONFIG_DIGIC_IV
#define RAW_PHOTO_EDMAC 0xc0f04208
#endif

#ifdef CONFIG_DIGIC_V
#define RAW_PHOTO_EDMAC 0xc0f04008
#endif

Just a suggestion that might simplify getting other Digic IV cameras up to speed with the great work reddeercity has been doing on the 5D2.

BTW--the "fixme" comment in propvalues.c probably still applies for Digic 4+ cameras like the 1300D.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: stokis on February 01, 2019, 10:43:50 PM
I tested latest 31.jan update on 3008x1074 @ 23.98 fps.
Now the motion is ok, as 23.98 fps should be. I can record ~18 sec without dropped frames.
And mlv sound also goes one on one with picture.

I hope you can somehow make crop preset with increased vertical resolution more than 1074 :)

Thank you reddeercity for your work!
I can use my old canon 5D mkII for filming my kids in 3k now :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on February 02, 2019, 01:58:44 AM
Curious about 30 fps in the Canon menu - why is this necessary? I tried it today with 24 and 30 and didn't notice any difference in my results.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 02, 2019, 06:35:55 AM
New Update: 3008x1080 @ 23.98 fps
magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip)

Update Notes:
Pushing the vertical without having to code CMOS[1] offset plus more user friendly preview.

I added 6 more vertical lines to 1080 from 1074 , this took some time to get right .
This is a far as I can go vertically without changing CMOS[1] offset valve .
Also you no longer need to select ML "gray scale" (B/W) preview ,
Just select "Auto Preview" and magic lantern will do the rest  :))
You still need to press half shutter for color preview  but it acts just like non crop_rec .
HDMI should be still working .

FYI on Head Timers : if you adjust both 3 & 4 head timers to the same vertical raw size + the offset
you get more fps - e.g. on 5d2 in 5x zoom (1080+52-offset =1132) =>0x46c
this is still below the default of 0x476 & 0x49C which canon uses for 1074
seem there put too much overhead in there for some reason.

@ jpegmasterjesse , I know it works either way but just for safety reasons .
It is bleeding edge . ;)
   
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on February 02, 2019, 08:10:31 AM
@reddeercity - Could you please commit your local changes and push to the remote?

You are working on this repository, right?

https://bitbucket.org/reddeercity/magic-lantern_10-12bit_raw/commits/all
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on February 02, 2019, 06:47:44 PM
@reddeercity - Could you please commit your local changes and push to the remote?

Great call, @dfort!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 02, 2019, 07:51:46 PM
I work locally only , so until I'm done .......
Read though my post and you will see all the changes
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on February 02, 2019, 11:14:14 PM
Open source projects like these are never "done."

The last time you posted your local repository was in August (https://www.magiclantern.fm/forum/index.php?topic=18443.msg205303#msg205303) and from the looks of it, you were working from a clone of my repository.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4899%2F39998071683_42595a6fea_c.jpg&hash=e41ca8605daa6452725f0f6ffdfa48e4) (https://flic.kr/p/23Wuzcz)

Hey, not a problem. I'm glad that you're building off some of the stuff that I pushed out there. If you want to continue working locally only, that's your choice. I'm just asking that you please reciprocate.

By the way, your binaries show the uncommitted changes in plain view. Opening your latest autoexec.bin in a text editor reveals this (I masked out your email address):

Code: [Select]
Magic Lantern crop_rec-4k-5D2-eXperimental.2019Feb01.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 8bfbb0ca228e+ (crop_rec_4k_5D2) tip
Built on : 2019-02-02 04:40:34 by david@***

diff -r 8bfbb0ca228e src/raw.c
--- a/src/raw.c Sat Sep 08 11:26:34 2018 +0200
+++ b/src/raw.c Fri Feb 01 21:40:35 2019 -0700
@@ -32,6 +32,11 @@
 #include "fps.h"
 #include "platform/state-object.h"
 
+#ifdef CONFIG_EDMAC_RAW_PATCH
+#include "patch.h"
+#include "edmac.h"
+#endif
+
 #undef RAW_DEBUG        /* define it to help with porting */
 #undef RAW_DEBUG_DUMP   /* if you want to save the raw image buffer and the DNG from here */
 #undef RAW_DEBUG_BLACK  /* for checking black level calibration */
@@ -221,7 +226,7 @@
 
 #define SHAD_GAIN_REGISTER 0xC0F08030
 
-static int lv_raw_type = PREFERRED_RAW_TYPE;
+//static int lv_raw_type = PREFERRED_RAW_TYPE;//
 static int lv_raw_gain = 0;
 
 /**
@@ -1831,6 +1836,10 @@
     #endif
 }
 
+#if defined(CONFIG_EDMAC_RAW_SLURP) || defined(CONFIG_EDMAC_RAW_PATCH)
+static int lv_raw_type = PREFERRED_RAW_TYPE;
+#endif
+
 #ifdef CONFIG_EDMAC_RAW_SLURP
 
 void FAST raw_lv_vsync()
@@ -1867,7 +1876,31 @@
     /* overriding the buffer is only valid for one frame */
     redirected_raw_buffer = 0;
 }
+#endif
 
+#ifdef CONFIG_EDMAC_RAW_PATCH
+static void raw_lv_setedmac_patch(uint32_t* regs, uint32_t* stack, uint32_t pc)
+{
+    /* R0: EDMAC channel */
+    /* R1: output buffer */
+    /* R2: EDMAC info (geometry) */
+    /* R3: flags */
+
+   int width, height;
+   int ok = raw_lv_get_resolution(&width, &height);
+   if (ok)
+   {
+      /* update EDMAC image size */
+      int pitch = width * raw_info.bits_per_pixel / 8;
+      static struct edmac_info dst_edmac_info;
+      dst_edmac_info.xb = pitch;
+      dst_edmac_info.yb = height - 1;
+      regs[2] = (uint32_t) &dst_edmac_info;
+
+     /* we can override this here */
+     EngDrvOut(RAW_TYPE_REGISTER, lv_raw_type);
+  }
+}
 /* integer gain used to fix the image darkening caused by lv_raw_gain */
 /* this gain must not (!) change the raw data */
 int _raw_lv_get_iso_post_gain()
@@ -2158,6 +2191,12 @@
 }
 
 #ifdef CONFIG_RAW_LIVEVIEW
+
+#ifdef CONFIG_EDMAC_RAW_PATCH
+extern thunk StartImagePass_x1_SetEDmac;
+extern thunk StartImagePass_x5_SetEDmac;
+#endif
+
 static void raw_lv_enable()
 {
     /* make sure LiveView is fully started before enabling the raw flag */
@@ -2168,6 +2207,10 @@
 
 #ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 1);
+#ifdef CONFIG_EDMAC_RAW_PATCH
+   patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x1");
+   patch_hook_function((uint32_t) &StartImagePass_x5_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x5");
+#endif
 #endif
 
 #ifdef DEFAULT_RAW_BUFFER
@@ -2210,6 +2253,10 @@
 
 #ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 0);
+#ifdef CONFIG_EDMAC_RAW_PATCH
+   unpatch_memory((uint32_t) &StartImagePass_x1_SetEDmac);
+   unpatch_memory((uint32_t) &StartImagePass_x5_SetEDmac);
+#endif
 #endif
 
 #ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
diff -r 8bfbb0ca228e platform/5D2.212/internals.h
--- a/platform/5D2.212/internals.h Sat Sep 08 11:26:34 2018 +0200
+++ b/platform/5D2.212/internals.h Fri Feb 01 21:40:35 2019 -0700
@@ -131,6 +131,9 @@
 #define CONFIG_RAW_DISABLE_IN_10X_ZOOM
 #define CONFIG_RAW_DISABLE_IN_10X_ZOOM_WEAK
 
+/** experimental - patch raw buffer parameters in Canon code */
+#define CONFIG_EDMAC_RAW_PATCH
+
 /** Use joystick for one-finger menu navigation */
 #define CONFIG_LONG_PRESS_JOYSTICK_MENU
 
diff -r 8bfbb0ca228e platform/5D2.212/stubs.S
--- a/platform/5D2.212/stubs.S Sat Sep 08 11:26:34 2018 +0200
+++ b/platform/5D2.212/stubs.S Fri Feb 01 21:40:35 2019 -0700
@@ -222,6 +222,10 @@
 NSTUB(0xFF9D8250,  DispSensorStart)
 NSTUB(0xFF9B1BE0,  LightMeasure_n_Callback_r0)
 
+/** LiveView RAW patches **/
+NSTUB(0xFFA08008, StartImagePass_x1_SetEDmac)               /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
+NSTUB(0xFFA08B24, StartImagePass_x5_SetEDmac)               /* FIXME: why it fails when the hook is placed on the BL instruction?! */
+
 /** Making the card bootable **/
 NSTUB(   0x20890,  cf_device)
 NSTUB(   0x208D8,  sd_device)

Running the module_hginfo_dump.sh script on your compiled modules also show your uncommitted changes:

Code: [Select]
./module_hginfo_dump.sh /Users/rosiefort/Downloads/magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212/ML/modules/crop_rec.mo
Name        : Crop mode recording
Author      : a1ex
License     : GPL
Summary     : Turn the 1080p and 720p video modes into 1:1 sensor crop modes
Description : This alters the 1080p and 720p video modes, transforming them
              into 3x (1:1) crop modes, by tweaking the sensor registers.
               [...]
Last update : 8bfbb0c on 2018-09-08 09:26:34 UTC by waza57:
              now we can change the ISO settings in crop x5
Build date  : 2019-01-27 21:31:15 UTC
Build user  : david@***

modules/crop_rec/Makefile
modules/crop_rec/README.rst
modules/crop_rec/crop_rec.c
diff -r 8bfbb0ca228e modules/crop_rec/crop_rec.c
--- a/modules/crop_rec/crop_rec.c
+++ b/modules/crop_rec/crop_rec.c
@@ -187,10 +187,10 @@
 static uint32_t MEM_ADTG_WRITE  = 0;
 static uint32_t ENGIO_WRITE     = 0;
 static uint32_t MEM_ENGIO_WRITE = 0;
-static uint32_t RAWROMHEIGHT_WRITE = 0;
-static uint32_t RAWROMWIDTH_WRITE = 0;
-static uint32_t MEM_RAWROMHEIGHT = 0;
-static uint32_t MEM_RAWROMWIDTH = 0;         
+//static uint32_t RAWROMHEIGHT_WRITE = 0;
+//static uint32_t RAWROMWIDTH_WRITE = 0;
+//static uint32_t MEM_RAWROMHEIGHT = 0;
+//static uint32_t MEM_RAWROMWIDTH = 0;         
 
 /* from SENSOR_TIMING_TABLE (fps-engio.c) or FPS override submenu */
 static int fps_main_clock = 0;
@@ -343,10 +343,10 @@
         case CROP_PRESET_CENTER_Z:
               if (is_5D2)                       
             {
-              skip_left       = 264;    //265 269bad value create "black meean too..." and bug in hack suite
-              skip_right      = 44;
-              skip_top        = 54;      // waza57 pourquoi 54 pour avoir une image correcte alors que dmspy que 2
-              skip_bottom     = 66;        //52
+              skip_left       = 160;    //265 269bad value create "black meean too..." and bug in hack suite
+              skip_right      = 0;
+              skip_top        = 52;      // waza57 pourquoi 54 pour avoir une image correcte alors que dmspy que 2
+              skip_bottom     = 0;        //52
             }
             
            break;
@@ -591,7 +591,7 @@
                 }
                 if (is_5D2)
                 {
-                cmos_new[1] = PACK12(42,57);//0xBB3;    /* pink highlights without this */
+                cmos_new[1] = PACK12(42,57);  //0xBB3; /* pink highlights without this */
                 cmos_new[2] = 0x10E;    /* read every column, centered crop */     //waza57 cmos 6
                 break;
                 }
@@ -685,10 +685,10 @@
             /* raw buffer centered in zoom mode */
             case CROP_PRESET_CENTER_Z:
                 cmos_new[1] = (!is_5D2)
-                  ? PACK12(9+2,42+1) /* vertical (first|last) */
+                  ?  PACK12(9+2,42+1) /* vertical (first|last) */
                   : -1;
                 cmos_new[2] = (!is_5D2)
-                ? 0x09E            /* horizontal offset (mask 0xFF0) */
+                ? 0x9E            /* horizontal offset (mask 0xFF0) */
                 : -1;
                 break;
         }
@@ -1510,8 +1510,8 @@
     /* attempt to reconfigure the x5 zoom at the FPS selected in Canon menu */
     if (is_5D2)
     {
-     int timerA = 804 ;         //waza57 good values for 23.976: 804- 1245  and 807 - 1237
-     int timerB = 1245 ;     
+     int timerA = 834 ;         //waza57 good values for 23.976: 804- 1245  and 807 - 1237
+     int timerB = 1200 ;     
      int a = reg_override_fps_nocheck(reg, timerA, timerB, old_val);
         if (a) return a;
            switch (reg)
@@ -1520,11 +1520,32 @@
                 /* X: (3072+140)/8 + 0x17, adjusted for 3072 in raw_rec */
                 
                  case 0xC0F06084:
-                    return (old_val & 0x0000000)  ;
+                    return (old_val & 0x0000000)  + 0x30036 ;
                 
                 case 0xC0F06088:
                     //return (old_val & 0xFFFF0000) + 0x630 ;
-                    return (old_val & 0x00000000) + 0x4b0063c;   //0x4b00648 or origin value chang nothing               
+                    return (old_val & 0x00000000) + 0x46F066A;   //0x4b00648 or origin value chang nothing
+
+                   /* HEAD3 timer */
+                   /* 467 in 30p */
+                case 0xc0f0713c:
+                   return 0x46C ; 
+
+                   /* HEAD4 timer */
+                   /* 49C in 30p */
+                case 0xC0F07150:
+                   return 0x46C ;
+
+                case 0xC0F08184:
+                   return 0x46C ;
+
+                case 0xC0F08188:
+                   return 0xC67 ;
+
+                case 0xC0F08518:
+                   return 0x46C0C67 ;
+
+
             }
     }
     else
@@ -1640,25 +1661,25 @@
 
 static int patch_active = 0;
 static int lv_dirty = 1; //waza57 changes to effective update for 5d2
-static void update_patch(int ROMhack) 
+static void update_patch() 
 {
     if (CROP_PRESET_MENU && !patch_active)
     {
         /* update preset */
         crop_preset = CROP_PRESET_MENU;
-        if (is_5D2 && crop_preset == CROP_PRESET_CENTER_Z )       // need to patch ROM for increase height for 5D2
-        {
-            patch_memory(RAWROMWIDTH_WRITE, MEM_RAWROMWIDTH, 0x00001C70, "crop_rec: rawromwidth hook");                                                                                                                   
-            patch_memory(RAWROMHEIGHT_WRITE, MEM_RAWROMHEIGHT, 0x00000767, "crop_rec: rawromwidth hook");             
-            if (ROMhack)
-              {
-                 patch_active = 1;
-                 return;                                                                                     
-              }
-        }                                                                               
+        //if (is_5D2 && crop_preset == CROP_PRESET_CENTER_Z )       // need to patch ROM for increase height for 5D2
+       // {
+            //patch_memory(RAWROMWIDTH_WRITE, MEM_RAWROMWIDTH, 0x00001C70, "crop_rec: rawromwidth hook");                                                                                                                   
+            //patch_memory(RAWROMHEIGHT_WRITE, MEM_RAWROMHEIGHT, 0x00000767, "crop_rec: rawromwidth hook");             
+            //if (ROMhack)
+              //{
+                 //patch_active = 1;
+                 //return;                                                                                     
+              //}
+        //}                                                                               
         /* install our hooks, if we haven't already do so */
-      //  if (!patch_active)   waza57 changes to effective update for 5d2
-      //  {
+         if (!patch_active)   //waza57 changes to effective update for 5d2
+        {
             patch_hook_function(CMOS_WRITE, MEM_CMOS_WRITE, &cmos_hook, "crop_rec: CMOS[1,2,6] parameters hook");
             patch_hook_function(ADTG_WRITE, MEM_ADTG_WRITE, &adtg_hook, "crop_rec: ADTG[8000,8806] parameters hook");
             if (ENGIO_WRITE)
@@ -1666,8 +1687,8 @@
                 patch_hook_function(ENGIO_WRITE, MEM_ENGIO_WRITE, engio_write_hook, "crop_rec: video timers hook");
             }
             patch_active = 1;
-            //return ; //waza57 present in my version not in  standard
-      //  }
+            return ; //waza57 present in my version not in  standard
+        }
       
     }
     else
@@ -1675,16 +1696,16 @@
         /* undo active patches, if any */
         if (patch_active)
         {
-           if (is_5D2 && crop_preset == CROP_PRESET_CENTER_Z)
-            {
-              unpatch_memory(RAWROMHEIGHT_WRITE);           
-              unpatch_memory (RAWROMWIDTH_WRITE);
-              if (ROMhack)
-              {   
-                  patch_active = 0;
-                  return;
-              }
-            }
+           //if (is_5D2 && crop_preset == CROP_PRESET_CENTER_Z)
+            //{
+              //unpatch_memory(RAWROMHEIGHT_WRITE);           
+              //unpatch_memory (RAWROMWIDTH_WRITE);
+              //if (ROMhack)
+              //{   
+                  //patch_active = 0;
+                  //return;
+              //}
+            //}
             unpatch_memory(CMOS_WRITE);
             unpatch_memory(ADTG_WRITE);
             if (ENGIO_WRITE)
@@ -2310,11 +2331,11 @@
         ENGIO_WRITE = 0xFF9A5618;
         MEM_ENGIO_WRITE = 0xE92D407C;
         
-        RAWROMHEIGHT_WRITE = 0xFFCAA1F4;
-        MEM_RAWROMHEIGHT = 0x00000467;
+       // RAWROMHEIGHT_WRITE = 0xFFCAA1F4;
+        //MEM_RAWROMHEIGHT = 0x00000467;
         
-        RAWROMWIDTH_WRITE = 0xFFCAA1EC;   // waza57 patch ROM for more height
-        MEM_RAWROMWIDTH = 0x00000FCE;
+        //RAWROMWIDTH_WRITE = 0xFFCAA1EC;   // waza57 patch ROM for more height
+        //MEM_RAWROMWIDTH = 0x00000FCE;
         is_5D2 = 1;
         
         crop_presets                = crop_presets_5d2;

Hum--looks like you are working off of the repository that waza57 updated back in September (https://bitbucket.org/waza57/magic-lantern/commits/all). So using the tools built into ML we can reverse engineer your local repository--or you can do us a big favor and:

hg commit
hg push


You are doing some great work that will benefit all Digic IV ML users. Keep it up!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on February 02, 2019, 11:54:12 PM
dfort is right

I was also going to say you should make the source code easily accessible. So we don't have to add changes by looking trough your posts. What you're doing may also be breaking the GPL license, it states that you should give someone the code if you give them the software. But it's not the end of the world yet.

At least upload the source in a zip! if you really don't want to mess around with installing mercurial, which is very easy btw. Would make us all a lot happier and I don't mean this to sound mean with this post :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 03, 2019, 03:06:43 AM
If any of you would look at my downloads page you will see the source code I use , It been there from the start !
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-50d-magic-lantern_1-9-2019.zip
I use waza57 source (and in my posts I state this , if anyone would have cared to read a few)
I have stated many times I work locally and don't work online as I don't know anything about bitbucket , of which I stated
many times also , sorry my work method are unconventional but that's the way I work & is the most comfortable for me .

So if it makes people feel better , I won't post code until I'm done all the presets in a working crop_rec module
Open source projects like these are never "done."
Well for me I'm almost "done" , after cf card over clocking & lossless I have no more goals .

Edit: updated first Post (https://www.magiclantern.fm/forum/index.php?topic=19336.msg182476#msg182476) with the new 3008x1080 23.98fps build & details
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on February 03, 2019, 03:19:04 AM
after you're done with the cf overclocking and lossless, it's time to grab a 5d4!!!

please!! :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 03, 2019, 06:15:01 AM
Fix the black level problem with 48fps in 3x3 1.67 squeezes (1856x688->1856x1146)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on February 03, 2019, 08:55:58 AM
...time to grab a 5d4!!!

And a beer! I'd suggest a strong Canadian brew like Unibroue La Fin du Monde.

So if it makes people feel better , I won't post code until I'm done all the presets in a working crop_rec module Well for me I'm almost "done" , after cf card over clocking & lossless I have no more goals .

Doesn't make me feel any better and having no more goals only happens when you're dead.

In any case, keep at it in whatever way works best for you. Just for fun, I downloaded that repository (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-50d-magic-lantern_1-9-2019.zip) that you said was there from the start and it has uncommitted changes. Yep, you cloned the waza57 repository (https://bitbucket.org/waza57/magic-lantern/src/unified/) and made these changes on the crop_rec_4k_5D2 branch (https://bitbucket.org/waza57/magic-lantern/branch/crop_rec_4k_5D2):

Code: [Select]
diff --git a/Makefile.user.default b/Makefile.user.default
--- a/Makefile.user.default
+++ b/Makefile.user.default
@@ -202,7 +202,7 @@
 # magiclantern-0.2.0.rc1.550d.fw109.zip
 #~ VERSION=0.2.0.rc1.550d.fw109
 BUILDVER=$(shell whoami).$(shell hg id -i -r .)
-VERSION = $(call eval_once,VERSION,Nightly.$(shell LC_TIME=EN date +'%Y%b%d').$(MODEL)$(FW_VERSION))
+VERSION = $(call eval_once,VERSION,crop_rec-4k-5D2-eXperimental.$(shell LC_TIME=EN date +'%Y%b%d').$(MODEL)$(FW_VERSION))
 
 # Build configuration - untested and debug stuff handy for devs
 # these can be specified either in Makefile.user, or in the command line:
diff --git a/modules/Makefile.modules.default b/modules/Makefile.modules.default
--- a/modules/Makefile.modules.default
+++ b/modules/Makefile.modules.default
@@ -19,4 +19,6 @@
  adv_int \
  crop_rec \
  edmac \

\ No newline at end of file
+        adtg_gui \
+        cf_acc \

diff --git a/modules/mlv_rec/mlv_rec.c b/modules/mlv_rec/mlv_rec.c
--- a/modules/mlv_rec/mlv_rec.c
+++ b/modules/mlv_rec/mlv_rec.c
@@ -133,8 +133,8 @@
  * => if my math is not broken, this traslates to resolution being multiple of 32 pixels horizontally
  * use roughly 10% increments
  **/
-static uint32_t resolution_presets_x[] = {  640,  960,  1280,  1600,  1920,  2240,  2560,  2880,  3200,  3520 };
-#define  RESOLUTION_CHOICES_X CHOICES(     "640","960","1280","1600","1920","2240","2560","2880","3200","3520")
+static uint32_t resolution_presets_x[] = {  640,  960,  1280,  1600,  1920,  2240,  2560,  2880,  3008,  3200,  3520,  3840,  4096,  5632 };
+#define  RESOLUTION_CHOICES_X CHOICES(     "640","960","1280","1600","1920","2240","2560","2880","3008","3200","3520","3840","4096","5632")
 
 static uint32_t aspect_ratio_presets_num[]      = {   5,    4,    3,       8,      25,     239,     235,      22,    2,     185,     16,    5,    3,    4,    12,    1175,    1,    1 };
 static uint32_t aspect_ratio_presets_den[]      = {   1,    1,    1,       3,      10,     100,     100,      10,    1,     100,      9,    3,    2,    3,    10,    1000,    1,    2 };
diff --git a/platform/50D.109/internals.h b/platform/50D.109/internals.h
--- a/platform/50D.109/internals.h
+++ b/platform/50D.109/internals.h
@@ -138,6 +138,9 @@
 #define CONFIG_RAW_DISABLE_IN_10X_ZOOM
 #define CONFIG_RAW_DISABLE_IN_10X_ZOOM_WEAK
 
+/** experimental - patch raw buffer parameters in Canon code */
+#define CONFIG_EDMAC_RAW_PATCH
+
 /** Use joystick for one-finger menu navigation */
 #define CONFIG_LONG_PRESS_JOYSTICK_MENU
 
diff --git a/platform/50D.109/stubs.S b/platform/50D.109/stubs.S
--- a/platform/50D.109/stubs.S
+++ b/platform/50D.109/stubs.S
@@ -231,6 +231,10 @@
 NSTUB(0xFF9ADD28,  DispSensorStart)                         // d
 // NSTUB(    ???,  LightMeasure_n_Callback_r0)              /* present on 7D.203, 5D2.212 */
 
+/** LiveView RAW patches **/
+NSTUB(0xff9db254, StartImagePass_x1_SetEDmac)               /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
+NSTUB(0xff9dbce8, StartImagePass_x5_SetEDmac)               /* FIXME: why it fails when the hook is placed on the BL instruction?! */
+
 /** Making the card bootable **/
 NSTUB(   0x1D6FC,  cf_device)                               // in cfReadBlk
 NSTUB(   0x1D744,  sd_device)                               // in sdReadBlk
diff --git a/platform/5D2.212/internals.h b/platform/5D2.212/internals.h
--- a/platform/5D2.212/internals.h
+++ b/platform/5D2.212/internals.h
@@ -131,6 +131,9 @@
 #define CONFIG_RAW_DISABLE_IN_10X_ZOOM
 #define CONFIG_RAW_DISABLE_IN_10X_ZOOM_WEAK
 
+/** experimental - patch raw buffer parameters in Canon code */
+#define CONFIG_EDMAC_RAW_PATCH
+
 /** Use joystick for one-finger menu navigation */
 #define CONFIG_LONG_PRESS_JOYSTICK_MENU
 
diff --git a/platform/5D2.212/stubs.S b/platform/5D2.212/stubs.S
--- a/platform/5D2.212/stubs.S
+++ b/platform/5D2.212/stubs.S
@@ -222,6 +222,10 @@
 NSTUB(0xFF9D8250,  DispSensorStart)
 NSTUB(0xFF9B1BE0,  LightMeasure_n_Callback_r0)
 
+/** LiveView RAW patches **/
+NSTUB(0xFFA08008, StartImagePass_x1_SetEDmac)               /* right before the SetEDmac call from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
+NSTUB(0xFFA08B24, StartImagePass_x5_SetEDmac)               /* FIXME: why it fails when the hook is placed on the BL instruction?! */
+
 /** Making the card bootable **/
 NSTUB(   0x20890,  cf_device)
 NSTUB(   0x208D8,  sd_device)
diff --git a/src/backtrace.c b/src/backtrace.c
--- a/src/backtrace.c
+++ b/src/backtrace.c
@@ -12,7 +12,7 @@
 #include "tasks.h"
 #include "backtrace.h"
 
-#define qemu_log_mask(...)
+#define qemu_log_mask(level, fmt, ...) { qprintf(fmt, ## __VA_ARGS__); }
 #define qemu_loglevel_mask(x) 0
 #define EOSState void
 
diff --git a/src/raw.c b/src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -32,6 +32,11 @@
 #include "fps.h"
 #include "platform/state-object.h"
 
+#ifdef CONFIG_EDMAC_RAW_PATCH
+#include "patch.h"
+#include "edmac.h"
+#endif
+
 #undef RAW_DEBUG        /* define it to help with porting */
 #undef RAW_DEBUG_DUMP   /* if you want to save the raw image buffer and the DNG from here */
 #undef RAW_DEBUG_BLACK  /* for checking black level calibration */
@@ -221,7 +226,7 @@
 
 #define SHAD_GAIN_REGISTER 0xC0F08030
 
-static int lv_raw_type = PREFERRED_RAW_TYPE;
+//static int lv_raw_type = PREFERRED_RAW_TYPE;//
 static int lv_raw_gain = 0;
 
 /**
@@ -1831,6 +1836,10 @@
     #endif
 }
 
+#if defined(CONFIG_EDMAC_RAW_SLURP) || defined(CONFIG_EDMAC_RAW_PATCH)
+static int lv_raw_type = PREFERRED_RAW_TYPE;
+#endif
+
 #ifdef CONFIG_EDMAC_RAW_SLURP
 
 void FAST raw_lv_vsync()
@@ -1867,7 +1876,31 @@
     /* overriding the buffer is only valid for one frame */
     redirected_raw_buffer = 0;
 }
+#endif
 
+#ifdef CONFIG_EDMAC_RAW_PATCH
+static void raw_lv_setedmac_patch(uint32_t* regs, uint32_t* stack, uint32_t pc)
+{
+    /* R0: EDMAC channel */
+    /* R1: output buffer */
+    /* R2: EDMAC info (geometry) */
+    /* R3: flags */
+
+   int width, height;
+   int ok = raw_lv_get_resolution(&width, &height);
+   if (ok)
+   {
+      /* update EDMAC image size */
+      int pitch = width * raw_info.bits_per_pixel / 8;
+      static struct edmac_info dst_edmac_info;
+      dst_edmac_info.xb = pitch;
+      dst_edmac_info.yb = height - 1;
+      regs[2] = (uint32_t) &dst_edmac_info;
+
+     /* we can override this here */
+     EngDrvOut(RAW_TYPE_REGISTER, lv_raw_type);
+  }
+}
 /* integer gain used to fix the image darkening caused by lv_raw_gain */
 /* this gain must not (!) change the raw data */
 int _raw_lv_get_iso_post_gain()
@@ -2158,6 +2191,12 @@
 }
 
 #ifdef CONFIG_RAW_LIVEVIEW
+
+#ifdef CONFIG_EDMAC_RAW_PATCH
+extern thunk StartImagePass_x1_SetEDmac;
+extern thunk StartImagePass_x5_SetEDmac;
+#endif
+
 static void raw_lv_enable()
 {
     /* make sure LiveView is fully started before enabling the raw flag */
@@ -2168,6 +2207,10 @@
 
 #ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 1);
+#ifdef CONFIG_EDMAC_RAW_PATCH
+   patch_hook_function((uint32_t) &StartImagePass_x1_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x1");
+   patch_hook_function((uint32_t) &StartImagePass_x5_SetEDmac, 0xE3A03202, raw_lv_setedmac_patch, "RAW LV x5");
+#endif
 #endif
 
 #ifdef DEFAULT_RAW_BUFFER
@@ -2210,6 +2253,10 @@
 
 #ifndef CONFIG_EDMAC_RAW_SLURP
     call("lv_save_raw", 0);
+#ifdef CONFIG_EDMAC_RAW_PATCH
+   unpatch_memory((uint32_t) &StartImagePass_x1_SetEDmac);
+   unpatch_memory((uint32_t) &StartImagePass_x5_SetEDmac);
+#endif
 #endif
 
 #ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER

BTW--I'm just sharing what I discovered about some tools included in the ML source code to look into ML binaries. I think it is possible to reconstruct your local repository provided you keep working the way you're working--and keep publishing test builds. Eventually you really should create a fork and make a pull request so you can get credit for your work (https://bitbucket.org/waza57/magic-lantern/annotate/8bfbb0ca228e11f091869df09a1b8f16dd36b508/modules/crop_rec/crop_rec.c?at=crop_rec_4k_5D2&fileviewer=file-view-default).
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on February 03, 2019, 09:03:58 AM
haha, dfort  :-*
Anyway. The code ought to be continuously published, gpl is all about collaboration.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Audionut on February 03, 2019, 10:52:37 AM
This project looks to be dangerously close to tragic lantern (https://www.magiclantern.fm/forum/index.php?topic=11080.0).

Is there any sort of commit history?

A dump of all of your code changes into a single commit, is likely to never ever be checked or pushed into the mainline.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on February 03, 2019, 08:45:06 PM
If any of you would look at my downloads page you will see the source code I use , It been there from the start !
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-50d-magic-lantern_1-9-2019.zip
I use waza57 source (and in my posts I state this , if anyone would have cared to read a few)
I have stated many times I work locally and don't work online as I don't know anything about bitbucket , of which I stated
many times also , sorry my work method are unconventional but that's the way I work & is the most comfortable for me .

This is good! But I downloaded the source you have uploaded(https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Crop_Rec-4k-5d2-50d-magic-lantern_1-9-2019.zip) and compiled it, it is only the 2880x1080 version. For any compiled version you release, you should also release the code specific to it. Could you upload a zip of the current 3008x1080 build's code please?

So if it makes people feel better , I won't post code until I'm done all the presets in a working crop_rec module Well for me I'm almost "done" , after cf card over clocking & lossless I have no more goals .

Definitely keep posting code, especially if you are uploading a compiled build. If you really don't want to use mercurial (hg), please just upload a zip file of your code every now and then. It will really make the ML developers a lot happier.

Ok , can you post your code please
I need to look at the code first before I try it ,
Don't what to brick my camera .

This
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Bender@arsch on February 19, 2019, 09:25:46 PM
The new Preset works great, but...

Is it possible to switch the live view on and off, as the recording time is reduced by ~1/4? Or implement the old one too?

And is it possible to change the area of the live view before recording? -> left middle position is not perfect sometimes. You can change it with the joyspad (during the recording), but only limited. Too much -> error
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 20, 2019, 06:46:14 AM
Is it possible to switch the live view on and off
No , that's how video works thought liveview . No liveview no video


... is it possible to change the area of the live view before recording? ......
No , that's the way it's in crop_rec.mo
If you want to move the crop window around  you must use the ML default 3x crop_mode without the crop_rec.mo enabled .

left middle position is not perfect sometimes.

That will be fixed in updated crop_rec.mo , to come very soon , just working out a few bugs right now.

Just some notes: for developers only
Figured out why/how to get canon liveview (real time)while recording raw with extended or reduced resolutions .
Simple ,  the final frame rate must be 23.976fps -- 23.98 , 24.004 or even below will give frozen liveview
I tested on a (1x3) Anamorphic 5k preset in 3xcrop I'm working on (1664x1842 @ 23.976fps = 4992x1842 )
If I chose canon liveview I get full canon color liveview in standbye or recording raw that's real time!
Plus this seems to fix HDMI , you no longer need to force HDMI -> VGA(480p)
Works now at 1080i , even when recording , but this does take resources from recording raw (less frames + the chance of corrupted frames)
I kept adjusting  A & B timer to fine the sweet spot , also noticed that at 23.976 the sensor run way cooler & liverview refresh is fast no delays
special panning the cam , everything just works smoother at 23.976.
I guess because the sensor runs at 24 MHz
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on February 20, 2019, 11:53:01 PM
Thank you so much, we are so waiting for it... Stupid question, but how to adjust that final framerate?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on February 21, 2019, 12:10:37 AM
Thank you so much, we are so waiting for it... Stupid question, but how to adjust that final framerate?

FPS overrride is not working with these builds as the timers need to be at special values... but you can hack it:

1. go in to crop mode
2. go to fps override, but do not turn it on
3. read the timer A and B values in advanced section
4. turn fps override on, and set the A and B values to what you read in step 3
5. increase timer A to reduce fps
6. (you may have to leave live view and go back to get rid of artifacts created while adjusting timers)
7. done

This is assuming you're using 5D2 and the 3k crop preset, as that is what worked for me
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on February 21, 2019, 05:33:39 AM
Sorry those note where intended for developers not end users
I was thinking out loud
I would not recommend adjusting timers a & b in crop_rec
can cause soft brick , or other possible issue/damage .
I'll have to disable that in future builds for end users

   
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on February 21, 2019, 04:57:59 PM
Okay, Thank you so much Ilia, actually, it's very interesting for me... But I think now I won't touch the timer values. Just in case will follow the advice of Reddeercity, because my 5DII is the only camera I have and I still work using it. But thank you anyway.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on February 21, 2019, 06:19:33 PM
Timer values can be altered to match crop rec and put into fps-engio.c but I don´t see why. Better to just profile them for a certain preset. I alter timers all the time for my eosm. Sometimes a tweak can lead to less corruption and so on. The thing is. If code would have been continuously updated I could have put in those tools myself but I assume code is coming one of these days...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 03, 2019, 05:35:40 PM
@reddeercity - Thanks for your notes on the registers for the 7D (https://www.magiclantern.fm/forum/index.php?topic=5601.msg212812#msg212812). I've got things pretty much working on a merge of the raw_video_10bit_12bit_LVState and RscMgr_memory branches so I pulled in the Digic Poke (https://magiclantern.fandom.com/wiki/Register_Map/Brute_Force) changes (https://bitbucket.org/waza57/magic-lantern/commits/db30a11dca7b9e531d69a74e0f5ff9947fe05068?at=crop_rec_4k_5D2) @waza57 made and it seems to be working.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7846%2F32325680587_90874d46a4.jpg&hash=a0212d2d2866f7e9aa8be5d9395e6729) (https://flic.kr/p/RfvyCt)

The default is to adjust 0xc0f08008 so I take it that this is an interesting register?

Code: [Select]
  0xc0f00000 Register family
+ 0x00008000 Register base
+ 0x00000008 Register offset
  --------------
  0xc0f08008 Value

Could you please give us a simple example of how to use this tool using something that you have already tested on the 5D2? I have a feeling that much of your work on that camera will translate to other Digic 4 cameras.

Thanks!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on March 03, 2019, 06:15:25 PM
Could you please give us a simple example of how to use this tool using something that you have already tested on the 5D2?

+1

I have a feeling that much of your work on that camera will translate to other Digic 4 cameras.

Likewise. Big time!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 04, 2019, 01:22:49 AM
.......
Could you please give us a simple example of how to use this tool using something that you have already tested on the 5D2? I have a feeling that much of your work on that camera will translate to other Digic 4 cameras.
Thanks!
Sure , have to dig it up & refresh my memory .

I'm slowly moving away from D4 (5d2) and moving in to D5 (5D3) for now (need the stable 4k raw for up coming projects)
So I'm actively looking (if any one knows of a good deal PM me).
If not then it's looking like a EOS R (4k 10bit canon log HDMI) or EOS RP(4k HDMI but in only 8bit & no Canon Log - $1000 cheaper)
I thought of a 5D4 but is so close to what the 5D3 can do internally and still it's only 1080p from the HDMI .
But I will keep my 5D2 for backup & further development's , still need to get Lossless & cf card bus overclocking working yet .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on March 06, 2019, 04:24:48 AM
Reddeercity, I understand you, but how sad it is... Do you still plan to finish your work with turning 5d2 into a UHD cinema camera and finalize your presets (including your awesome vertical squeezed preset) and lifeview?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 06, 2019, 05:24:20 AM
Reddeercity, I understand you, but how sad it is...
Do you still plan to finish your work with turning 5d2 into a UHD cinema camera and finalize your presets
(including your awesome vertical squeezed preset) and lifeview?
Yes , don't worry  ;) all that plus a surprise or two

....
(including your awesome vertical squeezed preset) and lifeview?
Oh ,  something farrrrr better ,
hint : has something to do with 5k  :P :P
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: honza123 on March 06, 2019, 10:48:45 AM
Great news !!
 :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on March 07, 2019, 03:31:58 AM
oh.... can't wait!!)) uh
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 07, 2019, 08:04:14 AM
A little preview of things to come ,
I just putting the finishing touches on 1x3 5k Anamorphic setting in 3xcrop_mode(5x zoom) (well not totally 5k, but  closer to 4.9K)  :D
Start with 1632x1828 @ 23.976 fps with less then 20us/line of rolling shutter , that's better FHD (around 24us/line)
Once you stretch the horizontal (3x) you get 4896x1828 @ 23.976 fps without any aliasing or moiré pattern  :o
That's not the best part ! I have real time liveview plus with corrected aspect ratio that not distorted !!

Before without any liveview A.R. Correction
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FtJDKcH3%2FSmall-No-N-Anamophic.png&hash=cef7644f622741a1c49a9e34e308a230) (https://imgbb.com/)

After with liveview A.R. Correction
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FmyFq2z3%2FSmall-Anamophic.png&hash=184876a8abac4cede350ef0fe8492b35) (https://imgbb.com/)

How did I do this you may ask ?
Well I just used the Anamorphic correction that's all ready there in magic lantern under display settings
Thou I may update that to 3:1 , which is the real corrected A.R. but it's close enough for now (that really cool)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FQXYfcg2%2FSmall-Anamophic-settings.png&hash=da4c0cc773f262452e97f765d1d4ee39) (https://imgbb.com/)>

Shot a quick video on my iPhone , showing this option being enabled and recording raw video .
I posted it on my google drive IMG_2710.MOV (https://drive.google.com/file/d/16EnP5D3kbbY4pHQO5zoJG7BFmA7WI-HI/view?usp=sharing)
Thou the correction does not work while recording raw video , it does work in standby mode only , great for framing you shot .
Also HDMI work great too , but correction does not work though the HDMI just on the LCD screen .

I'll try and push this one though for a test crop_rec module for tester , so keep a look out for it.

@dfort , I haven't forgot about you and the digital poke on 7D , just trying to wrap things up on the 5D2 crop_rec .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 07, 2019, 07:55:37 PM
Eager to test this out! Fantastic work!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: banertop on March 09, 2019, 12:25:19 AM
o man.....

this is not normal any more :) :) :) :) :) :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on March 09, 2019, 12:34:30 AM
Really exciting,

But it should be compared to the original '3.5K' preset (3008x1080), to make sure it actually gives a more detailed image. I'm still uncertain about 3x squeeze.

Can we see some samples maybe?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on March 09, 2019, 05:30:29 PM
Can we see some samples maybe?

+1
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 10, 2019, 02:14:08 AM
Ok , how this
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FrmTS4ZM%2F1x3-M07-0007-000000-small-320x358.png&hash=38f2d48351b472d534b69d7b4d905564) (https://imgbb.com/)
1x3_M07-0007_000000.png (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1x3_M07-0007_000000_Small_color_correction.png)
1x3_M07-0007_000000.dng (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/1x3_M07-0007_000000.dng)

Exported Tiff from Adobe CS6 A.E.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FnDHByK3%2F4896x1828-M007-0007-00000-small-490x183.png&hash=7c0204000de3006287b77506448534dd) (https://imgbb.com/)>
4896x1828_M007-0007_00000.tif (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4896x1828_M007-0007_00000.tif)

The only way to process this is in Adobe After Effect , non of the app on the form do 3x horizontal resolution I've only seen 2x , I did make a  request on the mlv app thread
but got no response . Beside that I like ACR anyways , I just set the project size e.g. 4896x1828 then just stretch out the horizontal , done .
Exported a ProRes 422LT ,  217MB 4896x1828_23.976fps_ProRes_422LT_M07-0007.mov (https://drive.google.com/file/d/1CNOXcG0fCuHkirdslmpN8Ffux1HG5iD3/view?usp=sharing)

Also here a mp4  from A.E.  , A.E. will not export at 4896x1828 h264 instead it exports at 4096x1828 , so it looks like it just crops the side .
4096x1828_A.E._cropped_M07-0007.mp4 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/4096x1828_A.E._cropped_M07-0007.mp4) 22MB
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on March 10, 2019, 02:50:00 AM
Thanks, actually looks much better than I expected! Can't say it's sharper than the 3008x1080 preset, but close, will be useful as it has less crop factor.

The only way to process this is in Adobe After Effect , non of the app on the form do 3x horizontal resolution I've only seen 2x , I did make a  request on the mlv app thread
but got no response .

You did get a response...

But now back to your request: 3.0x horizontal stretch is there since v1.5. If the file has the right metadata the setup works automatically. If not, you have to setup height stretch factor to 0.33x. MLVApp always upsamples since v1.5, so a height stretch factor of 0.33x is realized as width stretch factor of 3.0x. In older versions you can do this manually for ffmpeg export by entering your desired output resolution (frame is stretched once only on export, so no quality loss). Maybe we should add a information label somewhere, which shows the resolution after stretching.

Basically do this: set the vertical stretch factor to 0.33x in Transformation section, and MLV App will upsample horizontally by 3x.


I will admit that is confusing. @masc could we make it nicer and have it as horizontal 3.0x? Or even allow the user to enter custom value?


Also an shortened MLV would be nice if you could? to test out in MLV App :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 10, 2019, 07:06:49 AM
How large is the crop factor? I love the high resolution 5x stuff but the closer we can get to full frame with higher rez is obviously desirable.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: DeafEyeJedi on March 10, 2019, 07:49:18 AM
I will admit that is confusing. @masc could we make it nicer and have it as horizontal 3.0x? Or even allow the user to enter custom value?

Agreed. Good call.


Also an shortened MLV would be nice if you could? to test out in MLV App :)

+1
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 10, 2019, 11:14:28 AM
You did get a response...
Correct. (Works officially since december 2017 in MLVApp (v0.12), when entering resolution manually.)
I will admit that is confusing. @masc could we make it nicer and have it as horizontal 3.0x? Or even allow the user to enter custom value?
I already explained that somewhere else: this is close to impossible because we automate correct stretching in dependency to metadata. Now we set one UI element in dependency to this metadata. Doing it to horizontal stretching would make it far more difficult without advantage. Quite the contrary: when using 1x3 with horizontal stretching + anamorphic lens, it becomes impossible to stretch.
If I disable automation completely - no problem. But it won't be easier.
If it is easier for you, I could name the elements different: "anamorphic lens correction" (=horizontal stretching) and "sensor readout correction" or something like that (=vertical stretching), with 1x1 -> 1x1, 3x1 -> 1x1, 5x3 -> 1x1 and 1x3 -> 1x1.
On the other side, since a very long time in export options you can resize the export to any resolution without quality loss.

Edit:
I now implemented a label which shows what you'll get.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FNYtJ5t3%2FBildschirmfoto-2019-03-10-um-11-39-30.png&hash=17bda293e7cd28847d1c38a716ea9ae7)

Tip: if you set metadata in camera correctly, MLVApp shows the right AR automatically, and you also can fast export to DNG with correct AR. Adobe programs show correct AR out of the box then.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on March 10, 2019, 01:38:19 PM
Edit:
I now implemented a label which shows what you'll get.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FNYtJ5t3%2FBildschirmfoto-2019-03-10-um-11-39-30.png&hash=17bda293e7cd28847d1c38a716ea9ae7)

Tip: if you set metadata in camera correctly, MLVApp shows the right AR automatically, and you also can fast export to DNG with correct AR. Adobe programs show correct AR out of the box then.

Thanks! I like this

How large is the crop factor? I love the high resolution 5x stuff but the closer we can get to full frame with higher rez is obviously desirable.

Divide 5616 (sensor width) by the video width, for example: 5616/3008 = 1.86x crop

@reddeercity, I know you have been increasing the width mostly, but would it be possible to increase the height and ignore width? something like a 2000x1500 preset for example, instead of 3008x1080. It would be nice for anamorphic lenses or people who just enjoy aspect ratios.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 11, 2019, 04:54:45 AM
@ Ilia3101 & masc
Just tested mlv app V1.5 on windows , seem to work now , but I sure it didn't work before
that why ask on your thread . I don't get the same Info that you posted @ masc  when I select 0.33x .
That not on my windows version and I guess that the confusion right there , If I would have seen the
pixel stretch final resolution , I would have used mlv app .
Tip: if you set metadata in camera correctly, MLVApp shows the right AR automatically, and you also can fast export to DNG with correct AR. Adobe programs show correct AR out of the box then.
 
No it doesn't , at least on windows -- I can't get it to export more then 1 dng (only works with export frame) "fast export to DNG" does nothing
and the dng that's does export with "single frame export with 0.33x set is only 1632x609 not 4896x1828 , that only
happens with compressed file format like "PNG" or "H264" etc. ..... no pass though cdng's to 4.9k etc. ....

Even MLVFS with Quick Mount  ( which I use 99%  in After Effect & Resolve) doesn't show corrected a.r.
I guess as long as I have 16bit tiff's I'm happy , would like to have the right sized cdng (4896x1828) to work with .
Maybe mlv_dump can export them correctly never looked in to it yet .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 11, 2019, 06:58:26 AM
Can we expect to see a build to test soon? Are there any specific kinks you're still working out?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 11, 2019, 10:49:31 AM
@reddeercity:
The MLV and DNG code is identical on Win/Linux/OSX. But I tested on Windows now. Unfortunately I can't find a bug. I tried with a 1x3 MLV file from EOS-M (which has correct stretching metadata) and it works (AR is correct in viewer and in all possible DNG (single compressed/uncompressed, batch uncompressed, compressed, fast) / PNG exports without changing one single parameter in MLVApp).
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M19-1215.MLV
I opened the DNGs / PNGs in LR6.14 for testing. In my case I got always 3552x2000 from originally 1184x2000. You'll get something else if you override resolution by enabling "resize" box in export settings.

How did you test? Would be interesting - maybe somewhere are still bugs (I am sure there are).
If you like to see the new resolution label from my screenshot you should compile the latest commit - just pushed the code to the repos for now. But all the resizing should work in v1.5 as described.

The metadata which is used for correct AR are:
RAWC.binning_x, RAWC.skipping_x, RAWC.binning_y, RAWC.skipping_y

Somehow Resolve can't interpret stretching metadata - so here you'll get wrong AR out of the box.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: OlRivrRat on March 12, 2019, 06:04:31 AM
                     jpegmasterjesse

       If You Read the last 100 or so replies, I suspect You will find what You are Wanting ~
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on March 12, 2019, 01:37:51 PM
masc,
I have had the occasional mlv from my 5d3 that doesn't get automagically stretched.  It may have been only once.  I just recall it happening about a week or two ago.
Unfortunately I don't have the mlv's, as it was during my ongoing testing, and those files get ditched when I format the card.
I'll keep an eye out for it and if it happens again I'll send you the file...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Danne on March 12, 2019, 02:03:03 PM
@reddeercity:
How did you test? Would be interesting - maybe somewhere are still bugs (I am sure there are).
If you like to see the new resolution label from my screenshot you should compile the latest commit - just pushed the code to the repos for now. But all the resizing should work in v1.5 as described.
No bug. metadata has to be set in code, so where´s the code?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on March 12, 2019, 10:41:49 PM
No bug. metadata has to be set in code, so where´s the code?
+1

@reddeercity if you released code when you come up with more changes, all of your presets would be added to the menu by now, and the camera would set stretch metadata correctly. Danne is very keen on helping.

I appreciate that you have the original code uploaded, but it's important to have modified newer versions too.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 14, 2019, 04:04:41 AM
There's no new code yet , I'm doing everything though adtg_gui_.mo
Now that should clear up things about code etc. ....
and the 1x3 files display correctly in mlv app when exported to compressed format
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: ilia3101 on March 14, 2019, 09:51:49 PM
There's no new code yet , I'm doing everything though adtg_gui_.mo
Now that should clear up things about code etc. ....

Thanks it does clear stuff up. Then could we just have the code for the build that can do 3008x1080 resolution? Last time I checked, the uploaded code was the 2880x1080 version.

and the 1x3 files display correctly in mlv app when exported to compressed format

Good to hear it works.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 14, 2019, 10:02:22 PM
Then could we just have the code for the build that can do 3008x1080 resolution?
Have you tried this one from first post? https://www.magiclantern.fm/forum/index.php?topic=19336.msg182476#msg182476
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on March 16, 2019, 01:06:33 AM
confirmed here, uncompressed doesn't display correct aspect ratio automagically in mlvapp.  lossless does.
i remember now how it happened here.  i was troubleshooting a shot a while back and tried uncompressed.  i was focused on the issue and didn't pay much attention to the whole stretching thing...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 16, 2019, 09:52:59 AM
I don't think it has something to do with compression, because these metadata variables are always there. If they are used by ML in camera, MLVApp can use them as well, if not then not. But if you have such a file I could have a look.

Edit: quick test with EOS-M footage 10bit lossless vs. uncompressed: no difference in those parameters and behaviour.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on March 16, 2019, 01:14:48 PM
i'll PM you the file...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: kwstas on March 16, 2019, 02:02:44 PM
Hi reddeercity and guys,

I've made some quick test with experimental release. Followed the steps from p1. I'm facing these issues.

- 5x mode is magenta, when recording turns grayscale, yet crop factor is irrelevant to what it records
- when pressing to 10x mode camera freezes

And a hypothetical scenario.
Just wondering if it possible when 5x mode pressed the viewfinder remains in 1x with crop lines displayed based on chosen ratio.

cheers,
K.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 16, 2019, 07:30:13 PM
i'll PM you the file...
Thank you. The metadata is definitively wrong in your file - so it is not set correct in camera. This is how your file looks inside (look at the very right, I marked metadata blue):
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FWkxxwRD%2FBildschirmfoto-2019-03-16-um-19-26-19.png&hash=7f5cfe42295e591d47ea389e1ab3d039)
Your file is 1x3, but 3x3 is set in the MLV.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on March 16, 2019, 09:39:08 PM
as soon as i switch to lossless in camera, there's no problem.
uncompressed only.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: mlfan on March 17, 2019, 03:33:57 AM
Hi reddeercity and guys,

I've made some quick test with experimental release. Followed the steps from p1. I'm facing these issues.

- 5x mode is magenta, when recording turns grayscale, yet crop factor is irrelevant to what it records
- when pressing to 10x mode camera freezes

I'm a new 5D mk II user and I'm seeing this too. IIRC 10x mode doesn't always freeze, but I consistently have a very pink 5x mode.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 17, 2019, 05:01:30 AM
What preview mode are you using ?
remember this is bleeding edge stuff , sometimes things that
normally work on the nightly builds don't always work on bleeding edge stuff .
crop_rec is only meant to record extended resolutions in 3x crop_mode (5x zoom)
and not to be used with the regular nightly build feature set .
Meaning 10x zoom freezes the camera .
 
I expect to have normal canon previews in future releases of the crop_rec  in 3x crop_mode (5x Zoom)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: mlfan on March 18, 2019, 12:36:22 AM
What preview mode are you using ?
remember this is bleeding edge stuff , sometimes things that
normally work on the nightly builds don't always work on bleeding edge stuff .
crop_rec is only meant to record extended resolutions in 3x crop_mode (5x zoom)
and not to be used with the regular nightly build feature set .
Meaning 10x zoom freezes the camera .
 
I expect to have normal canon previews in future releases of the crop_rec  in 3x crop_mode (5x Zoom)

I'm not quite sure what you mean by preview mode -- I'm in manual mode in Live View. Could you explain?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on March 18, 2019, 03:19:46 AM
Please read first post of this thread with care. Esp. what is written about preview mode there.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 21, 2019, 05:29:45 AM
I've been experimenting around with the 7D to see if I can get it working with reddeercity's latest crop_rec settings. Still a long way off but I have managed to merge the current crop_rec_4k_mlv_snd branch with waza57's crop_rec_4k_5D2 along with raw_video_10bit_12bit_LVState -- but wait, there's more -- RscMgr_memory and the changes need to get the 7D working with 10/12bit. But that's not all, also the latest lua_fix. The 50D should be working on this branch. In fact all of ML cameras should be functional on this branch. Ok--they all compile but I haven't tested them. Need some help with that.

There's a test build for the 5D2 on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). It should work the same as the waza57 version. I didn't include reddeercity's unpublished changes at this time.

Here's the source code if you want to roll your own and start experimenting:

https://bitbucket.org/daniel_fort/magic-lantern/src/crop_rec_4k_mlv_snd_experiments/

@reddeercity -- I could use some tips from you on how to get started with Digic Poke to change the resolution.

[EDIT] Sorry, looks like I didn't merge waza57's code before I posted this.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 21, 2019, 05:58:32 AM
I don't really see why you merged in to "crop_rec_4k_mlv_snd branch"
I can't see any advantage with it , specially with lua_fix . That I can't see to be any use at all at least on 5d2 in Video .

I thinks it simpler & easier to keep on the waza57's crop_rec_4k_5D2 branch , in my own opinion .

@reddeercity -- I could use some tips from you on how to get started with Digic Poke to change the resolution.
Sorry , I been busy with getting a few projects up and running ( it spring time and warm weather finally has come to the great white north  :D)
and writing code for crop_rec 5d2 . I'll see if I can put something together in the next day of two .

Edit: just read though the crop_rec code and there's no code for waza57 crop_rec branch
There no reference to 5D2 in the code at all , so I would say it didn't merger any think for waza57 source "crop_rec_4k_5D2 " branch
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on March 21, 2019, 12:44:20 PM
I think it's better to work on a clean fork of crop_rec_4k_mlv_snd.

It will allow to prepare for merging back into the "official" codebase when things will be ready.

Regarding code from Waza, is it still necessary or a better way to allow digic4 to work on this branch was found?

I remember reddeercity has to reverse back some of the code to make it work as current state
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 21, 2019, 09:55:27 PM
I don't really see why you merged in to "crop_rec_4k_mlv_snd branch"

Why? Because it is there! Oh wait, that's a mountain climber's answer. Well, in a way this is quite a mountain to climb.

If you take a look at the experiments downloads page (https://builds.magiclantern.fm/experiments.html)--

4K raw video recording; lossless compression
crop_rec module with higher resolutions (4K, 1080p48 etc):
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7816%2F46521454375_88479116ab_z.jpg&hash=02b0c576ce9e65908fe52a9663bba4c4) (https://flic.kr/p/2dSWBAX)

I think it's better to work on a clean fork of crop_rec_4k_mlv_snd.

Yes, I did that. Basically the various development branches went into the Cuisinart and they blended together quite nicely with very few conflicts. Then I added some work I did go get the 7D working and voilà, a branch where the old 5D2 and 7D can live side by side with newer 5D3, 70D and the rest of the family too! @aprofiti - that means your 50D and eventually your manual_lens_info pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/940/manual-lens-info-support-to-crop_rec_4k/diff) should also be working on all cameras.

Ok--so much for the sales pitch, it is not quite ready for prime time. There's some build script problem that prevents "make zip" from working properly though "make install" works fine. Go figure. I'm also sure there are a few bugs that need to be addressed. Right now let's call it more a proof of concept.

Regarding code from Waza, is it still necessary or a better way to allow digic4 to work on this branch was found?

Edit: just read though the crop_rec code and there's no code for waza57 crop_rec branch
There no reference to 5D2 in the code at all , so I would say it didn't merger any think for waza57 source "crop_rec_4k_5D2 " branch

 :-X

Oops, that's the one branch that I haven't merged in yet. Thought I did it earlier. Need to resolve a few conflicts.

Well, it does answer the question whether Digic 4 can work on this branch but in order to get the crop_rec module working it does need some more work and since waza57 and reddeercity have already done that let's see if it can be merged in with the mainline experimental branches without breaking anything.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 22, 2019, 02:09:25 AM
Ok--still a little ruff around the edges but I got something that might work. Tested on the 7D and it does 10/12bit, the 50D should work the same. Now on the 5D2 I'm interested if the crop_rec module is working, it is from waza57's crop_rec_4k_5D2 branch. If it does work it would be interesting to see if reddeercity's crop_rec module works in this build.

Posted test builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Look for the builds from the magiclantern-crop_rec_4k_mlv_snd_experiments_waza57 branch.

@reddeercity - waza57's changes are showing up now:

https://bitbucket.org/daniel_fort/magic-lantern/commits/branch/crop_rec_4k_mlv_snd_experiments_waza57?page=5
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on March 22, 2019, 12:40:31 PM
Hello Dfort,

Been following all your posts with great excitement!  Thank you so much for not giving up on the fantastic 7D and trying to move forward with Digic IV cameras.

I have now briefly tested the build called:

magiclantern-crop_rec_4k_mlv_snd_experiments_waza57.2019Mar21.7D203

that I downloaded from your download area.  It works partially but in fact, worse than your March 2-nd build which works very well with all 3 bit depths with sound.  Here is what I found:

1)  I found only the mlv_rec.mo (RAW video (MLV)) in the modules directory.  MLV_Lite.mo is missing.

2)  Crop preview at all preview modes provides 5x and not 3x magnification.

3)  Once started, recording at all 3 bit depths does not stop upon pressing the rec button.  You get a red dot and a "stopping" message in the RAW video (MLV) menu and camera freezes.  Turning camera off and pulling battery out resolves the issue.

4)  In-camera playback does not work any more, not even after adding the raw_twk module.

5) Every mlv file gets recorded with 7 additional M00 through M07 files of 1 kB size each.  MLVApp does open and play the files though.  Sound works also.

Hoping that some of the other developers may be able to help you with useful hints, I keep my thumbs pressed for further progress on your part.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 22, 2019, 06:04:24 PM
Looks like it is one step forward and two steps back.

Main thing I'd like to find out is if the crop_rec module is working on the build I posted for the 5D2. Anyone?

The 7D has many issues. If I leave the battery in it overnight it wakes up dead--what I mean is that the battery drains even when the camera is turned off. I think this was reported before.

It works partially but in fact, worse than your March 2-nd build which works very well with all 3 bit depths with sound.

The March 2 build is strictly raw_video_10bit_12bit_LVState so if that's working nicely we should probably talk about how to integrate it into the main repository but let's continue with that topic on the 12-bit (and 10-bit) RAW video development discussion (https://www.magiclantern.fm/forum/index.php?topic=5601.new#new).

1)  I found only the mlv_rec.mo (RAW video (MLV)) in the modules directory.  MLV_Lite.mo is missing.

Right -- mlv_lite on the crop_rec_4k branch works with lossless compression so it won't even compile for Digic IV on my experimental branch. We're a ways off from getting that working on Digic 4. In the meantime most reports indicate that mlv_rec is working better than mlv_lite on Digic IV.

2)  Crop preview at all preview modes provides 5x and not 3x magnification.

Not sure what you mean by that. Maybe some pictures or video will help show the issue? I do have an issue when going into zoom mode I can't get out of it. Hey, this is highly experimental but I was able to record 233 frames of 2496x1198 10-bit at 24fps on the 7D.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7886%2F32500800867_7e7597371e.jpg&hash=51780b696dd7e4c33a2f1f56f46b3113) (https://flic.kr/p/RvZ6MD)

Ok--not the most exciting shot but it does work over here.

3)  Once started, recording at all 3 bit depths does not stop upon pressing the rec button.  You get a red dot and a "stopping" message in the RAW video (MLV) menu and camera freezes.  Turning camera off and pulling battery out resolves the issue.

Again--can't reproduce that issue over here.

4)  In-camera playback does not work any more, not even after adding the raw_twk module.

I never played around much with in-camera playback. @reddeercity's - do you have that working on the 5D2?

5) Every mlv file gets recorded with 7 additional M00 through M07 files of 1 kB size each.  MLVApp does open and play the files though.  Sound works also.

Strange, I'm not seeing that either. What happens with mlv_rec is that it often leaves an MLV_REC.TMP file on the card but it doesn't seem to have any negative consequences.

Hoping that some of the other developers may be able to help you with useful hints, I keep my thumbs pressed for further progress on your part.

That's why I'm posting on this topic. I'd like to get the crop_rec module working on the 7D (probably hard) and the 50D (probably easier) and maybe other Digic IV cameras.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dariSSight on March 22, 2019, 06:15:34 PM
It seems to freeze when in 3X mode and also have black lines vertical in the frame.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: aprofiti on March 22, 2019, 06:28:52 PM
I don't think I can experiments in the next days, but will follow and try something if possible.

Is the new branch cleaner compared to what you posted yesterday?
I mean only strictly necessaries (apart lua_fix which is easy to see) to understand difference between crop_rec_4k and 10-12 branch.

Can you make a "fake PR" to your bitbucket to evidence the difference with the main repository?

I need firstly understand what was changed with the code from Waza57 to make 5d2 works (I saw some commit which "revert" some previous one) and then start investigating on 50D (just " nearly standard" resolution because I'm stuck with incrementing heigth (https://www.magiclantern.fm/forum/index.php?topic=19336.msg210344#msg210344)).

Regarding compilation of ml_lite due to lossless.. can't remember if by adding the non definitive changes posted in lossless thread, will allow us to record raw videos... I was trying with silent pictures mainly.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: IDA_ML on March 22, 2019, 06:46:14 PM

The 7D has many issues. If I leave the battery in it overnight it wakes up dead--what I mean is that the battery drains even when the camera is turned off. I think this was reported before.

This was not the case with the March 02-nd build.  No battery drain here even after I left the battery for two weeks in the camera. Will check tonight if the battery gets drained with the March 22 build.

Not sure what you mean by that. Maybe some pictures or video will help show the issue? I do have an issue when going into zoom mode I can't get out of it. Hey, this is highly experimental but I was able to record 233 frames of 2496x1198 10-bit at 24fps on the 7D.

I mean that there is still no WySiWyG preview on today's build.  What you see on the screen while recording corresponds to 5x magnification while in fact the image is recorded at the correct 3x magnification.  5x-magnification does not allow for correct framing when you move the camera during recording.

Again--can't reproduce that issue over here.

Probably we have different camera settings.  Will keep testing tomorrow.  When I revert back to the March 02 build, recording is perfectly fine and stops nicely when I press the record button.  By the way, at 2496x1198 10-bit and 24fps, with the March 02 build, I get continuous recording on the 7D.  Please try that build again and compare it with today's one!

I never played around much with in-camera playback. @reddeercity's - do you have that working on the 5D2?

Please try the raw_twk.mo on your March 02 build.  Just add it to the Modules directory and activate it in the Modules menu.  It plays back all recorded files nicely in camera after they have been recorded. Very useful!

Strange, I'm not seeing that either. What happens with mlv_rec is that it often leaves an MLV_REC.TMP file on the card but it doesn't seem to have any negative consequences.

The  MLV_REC.TMP  always gets recorded on an empty card, no question about that.  I am talking about these 7 files (M00 through M07) short files that get recorded with every MLV file too. 

That's why I'm posting on this topic. I'd like to get the crop_rec module working on the 7D (probably hard) and the 50D (probably easier) and maybe other Digic IV cameras.

Can anyone help with this, please?  We badly need some help here and every hint that could guide us in the right direction will be greatly appreciated. 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 22, 2019, 08:12:45 PM
Now on the 5D2 I'm interested if the crop_rec module is working, it is from waza57's crop_rec_4k_5D2 branch. If it does work it would be interesting to see if reddeercity's crop_rec module works in this build.

I am getting a Raw Detect Error with 3.5k 5x mode on Crop_Rec. When adjusting values in the menu, at first it acted like it was in the 1920 1:1 mode, with 1984 as the max H resolution - after some bumbling around the menu now it expects to be able to get up to 2880x1080 - and I recorded a few seconds with accurate framing in liveview but frozen during recording. The black level is totally off in the clip, but it seems to be valid 2880 x 1080. (Uploaded the MLV here, M22-1357.MLV: https://drive.google.com/open?id=1vfodz-Dsk8T0NppBxFIQaCt59NSDEyZP)

And then, randomly, it now gives me a Raw Detect error again and I can get it to record again.

1920 1:1 works fine, with accurate framing until you press record. 12 bit has slightly incorrect black level, green tinted shadows.

Generally this build runs a little slow - the UI is slightly unresponsive here and there.

EDIT:
If it does work it would be interesting to see if reddeercity's crop_rec module works in this build.

It does. And much better. Adding the crop_rec module from reddeercity's magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212 to magiclantern-crop_rec_4k_mlv_snd_experiments_waza57.2019Mar21.5D2212 works like a charm.

Max resolution in crop_rec 3.5k 5x mode is 3008 x 1080. All bit depths work. Will keep messing around a little more.

Notes:
Sound Recording works, but after stopping recording the camera is stuck at "Stopping" for a long time - requires a restart but no battery pull necessary.

Is there a raw_twk module I could be trying on 5d2? MLV_PLAY hasn't worked for a long time with any of these 3k 5x builds, and the raw_twk I tried adding just gave me an oldAPI error.



Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 23, 2019, 01:52:16 AM
Ok--good to hear that reddeercity's crop_rec is working better. All hail to the 5D2 guru!

Merged in the unpublished reddeercity code which bumped everything up to a more current code base. This means that the lossless compression in mlv_play and some other places will be a problem until lossless is working on Digic IV cameras. Note that raw_twk does compile but I'm not sure what good it will do without mlv_play. Maybe start mixing and matching modules from other builds?

This time my camera did reproduce the issue where it wouldn't stop. Found a temporary fix for that.

src/raw.c
Code: [Select]
static void raw_lv_disable()
{
 // ASSERT(!lv_raw_gain); // dfort - 7D cannot stop recording if this is enabled
    lv_raw_enabled = 0;
    raw_info.buffer = 0;

Can you make a "fake PR" to your bitbucket to evidence the difference with the main repository?

Good idea, I was thinking about that myself. Here's a fake pull request (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/27/crop_rec_4k-for-digic-iv-cameras/diff) that shows the changes between my latest experimental branch and the crop_rec_4k_mlv_snd branch. Looks like lots of extra stuff that could probably be eliminated though it would still be huge and most likely not acceptable to merge into the main repository.

By the way, on these latest test builds adtg_gui should be working on the 50D so you should be able to start tweaking the registers to extend the resolution. On the 7D that should be possible using Digic Poke. (Once reddeercity wraps up some projects and takes a Spring break maybe he'll teach us how he uses that tool.)

Test builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

Is the new branch cleaner compared to what you posted yesterday?
I mean only strictly necessaries (apart lua_fix which is easy to see) to understand difference between crop_rec_4k and 10-12 branch.

As you can see from the fake pull request, it is getting complicated. The cleanest by far is the stuff I did on the raw_video_10bit_12bit_LVState branch but even there a PR with my changes to get the 7D working probably won't cut the mustard.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 23, 2019, 02:57:37 AM
Well it looks like I'm being pushed off my thread.
I have no intention of supporting  crop_rec_4k_mlv_snd branch on 5d2
I'll be finishing of the crop_rec in "crop_rec_4k_5D2" branch , Sorry it others disagree .
If there too many branches nothing will get done , so I will only help with development on that branch.

As for as other d4 go e.g. 7d,  go ahead and use "crop_rec_4k_mlv_snd branch" as I have no interest
in that camera or other d4's any more .

After I have finished code for 5d2 crop_rec then I may consider "crop_rec_4k_mlv_snd branch" but not before .
and that's a big maybe .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 23, 2019, 04:35:09 AM
I don't think anyone is trying to minimize your efforts - I'm very excited by everything you've added to the 5d2, and I'd hate to see development stall!

Could you explain the pros or cons between crop_rec_4k_mlv_snd /crop_rec_4k_5D2 in your opinion? When I added your crop_rec to crop_rec_4k_mlv_snd it was very stable and capable of all the same great stuff.

My understanding is that crop_rec_4k_mlv_snd more closely resembles development on other cameras and thus will be easier to keep the 5d2 up to date moving forward? It would be a shame if all your hard work was the last act for this legendary camera.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: theBilalFakhouri on March 23, 2019, 05:07:31 AM
Don't worry about the branches, crop_rec code will change completely to the new method of it, maybe at 1st April for this year for both D4 and D5.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 23, 2019, 05:27:56 AM
This time my camera did reproduce the issue where it wouldn't stop. Found a temporary fix for that.

src/raw.c
Code: [Select]
static void raw_lv_disable()
{
 // ASSERT(!lv_raw_gain); // dfort - 7D cannot stop recording if this is enabled
    lv_raw_enabled = 0;
    raw_info.buffer = 0;


This issue was still occurring for me on your latest test build - I'm assuming this code was included? This issue does not occur on reddeercity's Feb01 build.

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 23, 2019, 06:35:36 AM
Here is one of my first Test builds -- 2.6k crop_rec for the 5d2
2.6k_5d2_crop_rec.rar (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2.6k_5d2_crop_rec.rar)
I still have a vertical offset  problem in the code , it not applying cmos[1] correctly so there's a white bar at the bottom .
I'll work on it tomorrow , it's getting late .

So everything is working at 2624x1200 @ 23.976fps , realtime canon preview when preview set to "Auto"
It all works the same as nightly builds , HDMI works @ 480p the best , 1080i works but causes a odd flicker on screen
you can record raw video with HDMI connected . I quickly check that you can change shutter speed but didn't check ISO changes .
It should work on top of the build I have posted on the first page of this thread .

So check it out and let me know how it works please and yes there will be a white bar at the bottom of the image
that should be fixed tomorrow hopefully , I just want to know if everything else works correctly .

I will also be posting test crop_rec's for 48fps 3x3 & 1x3 anamorphic 4.9k @ 2.66 A.R. 23.976fps .
Once all that is tested and confirmed working then I'll have it all together in one "crop_rec"
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 23, 2019, 09:18:06 AM
Everything seemed normal! - besides what you mentioned.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: honza123 on March 23, 2019, 12:51:51 PM
I confirm the same.
 :)

There is a white bar (120 pixels) at the bottom of the image.
On the right side is a 16 point wide black bar.
The active area is 2608x1080 pixels.


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FhXbjY4Z%2FM23-1102-frame-172.png&hash=fea10806084b1b019eb2b9a83486924e) (https://ibb.co/hXbjY4Z)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 23, 2019, 05:53:09 PM
Hum--looks like I painted myself into a corner. Recording is still not stopping properly on my experimental branch:

This issue was still occurring for me on your latest test build - I'm assuming this code was included? This issue does not occur on reddeercity's Feb01 build.

There are some other issues too so I should either back up to find where the problem started, start all over or wait for reddeercity to publish his code.

Well it looks like I'm being pushed off my thread.

Didn't know you could own a thread on this forum. How about being positive and seeing this as continuing the discussion on a topic that you started?

I have no intention of supporting  crop_rec_4k_mlv_snd branch on 5d2

Fair enough, this is an open source project and we're all volunteers.

Quote
Every good work of software starts by scratching a developer's personal itch (https://opensource.com/article/17/4/itch-to-scratch-model-user-problems).
                                                                                       -- Eric Raymond

I only picked crop_rec_4k_mlv_snd because that seemed to be the right branch to eventually merge in your experimental work on the 5D2.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7816%2F46521454375_88479116ab_z.jpg&hash=02b0c576ce9e65908fe52a9663bba4c4) (https://flic.kr/p/2dSWBAX)

Now will your work eventually get merged in to this branch or another branch or will it become a new branch on the main repository? Something to think about. The worst-case scenario is that it doesn't get into the main repository at all.

I'll be finishing of the crop_rec in "crop_rec_4k_5D2" branch , Sorry it others disagree .
If there too many branches nothing will get done , so I will only help with development on that branch.

I would agree that there are too many open branches but I'm confused. It seems are you proposing a new branch only for the 5D2. Wouldn't it be better to have the 5D2 working on the same branch as the other ML supported cameras?

As for as other d4 go e.g. 7d,  go ahead and use "crop_rec_4k_mlv_snd branch" as I have no interest
in that camera or other d4's any more .

Sorry, I took your topic heading literally - 3K/UHD 5D2 Raw development and Other Digic IV Cams

Note that the crop_rec_4k branch at first only worked properly with the 5D3. Eventually the EOSM then other Digic V cameras were working on it. Maybe I'm the pushing to get the other Digic IV cameras started a little too early.

After I have finished code for 5d2 crop_rec then I may consider "crop_rec_4k_mlv_snd branch" but not before .
and that's a big maybe .

You know Magic Lantern is a work in progress and will probably never be "finished" -- right?  :)

As far as crop_rec_4k_mlv_snd and crop_rec_4k_mlv_lite_snd branches, those were started in order to get sound recording working with mlv_lite. At this point my understanding is that the Digic IV cameras are working better on mlv_rec. However, if we can get lossless compression working on Digic IV we will need to switch to the newer mlv_lite code.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 24, 2019, 12:55:08 AM
Not all that important, but I'm wondering if digital dolly works for any versions of crop_rec?  I think it'd be pretty great to scan vertically to get something like a full-sensor scan for landscapes. It hasn't worked for any of these builds we've been testing.

I noticed you can't move the preview window in 5x mode at all once digital dolly is enabled, wheras it does move without.

Here is what Alex said in another thread regarding digital dolly and crop_rec:

Modifying digital dolly to cover the full sensor is possible, but non-trivial (Canon recalibrates the vertical noise correction factors with each video mode reconfiguration, including when shifting the scanned area in x5 zoom mode). By default, Canon code scans a non-centered wide area in x5 zoom mode (3584x1320 usable area on 5D3) and moves it around as you move the focus box - notice a slight pause at certain positions on the screen, when it reconfigures the sensor.

You can, however, try something with the crop_rec_4k experimental build (caveat: requires fiddling)

- load crop_rec and mlv_lite
- set crop_rec preset to full-res LiveView
- reduce the vertical resolution in crop_rec submenu until you get the desired FPS (Target YRES = 1039 will give 23.973 FPS)
- recalibrate the CMOS[1] register to get good and centered image (trial and error)
- enable raw video and digital dolly - now it should cover the full width of the sensor.

If useful, I'll consider adding a full-width 1080p preset to crop_rec (maybe it can be pushed to about 5784x1080 at 23.976 FPS, or it might require slightly lower horizontal resolution).

Side note: the MLV spec also has metadata for 1-pixel increments for digital dolly, but would require special support from the MLV converter.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 24, 2019, 02:06:21 AM
Ok moving on .

Thanks to all tester  :)
I've removed that crop_rec from yesterday & updated with
2.6k_5D2_3-23-2019_crop_rec.rar (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2.6k_5D2_3-23-2019_crop_rec.rar)
Just use it with the build on the first page .
I fixed the cmos[1] reg in the code , now it works correctly with the full vertical height .
So now you have 2624x1200 @ 23.976fps with Canon liveview in real time !
There is about 10 or 12 line missing on the right side , I need to increase timer "A"
a little bit more , that I will take care of later on . So in reality horizontally ,  it's closer to 2605 or something like that .

Test it as if you are in a video production  :D
Let me know of any problems .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 24, 2019, 08:44:17 AM
Let me know of any problems .

Seems to work ok at all bit depths in non-crop and 5x crop. No issues with sound enabled.

I did try going to Canon 24p and it worked ok for a few shots and then I encountered issues with 5x preview mode being broken; on auto and on ML greyscale. A battery pull returned things to normal.Going in to 10x mode seems to trigger the same wonkiness no matter the Canon FPS, actually.

FPS override does not work in 5x mode.

Changing ISO's in 5x mode does work with the ISO button on the top of the camera. Changing the ISO in the ML menu with the joystick however, results in an infinite scroll where the ISO keeps increasing or decreasing depending on the direction you pushed. I believe this was also the case on crop_rec_4k_mlv_snd, but I will have to double check. Using the top scroll wheel in this menu does not cause this.

Using any method to change iso while in 5x zoom mode creates a white bar on the bottom of the frames of some ISOs - exiting to menu and back fixes this. Results for which ISOs are effected seems to depend on your starting ISO.

Gradual Exposure seems to be broken - there is no smooth transitions, and the briefly gets darker before brighter if you raise the iso

This is probably already known, but arrow/set shortcuts don't work in 5x mode.

ML digital ISO seems to have no effect with or without Raw modules loaded.

@dfort: One thing I'm noticing in your latest build magiclantern-crop_rec_4k_mlv_snd_experiments_waza57.2019Mar22.5D2212 is that there are 8 black pixels on the right side of the frame at 5x mode.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 25, 2019, 04:28:23 AM
Ok , thanks for testing .
All this is known issue , ISO on 5d2 works best anyways set to manual as I only use that.
I think I have a  fix for changing ISO in ML Menu.
10x Zoom is known to break liveview , Workaround : (in fact any liveview breakage)
Press play button twice (that refreshes the liveview ) (help if you have a h264 .mov or CR2 there to be reviewed)
That put you in 3x3 (FHD) Liveview should be ok now , press 5x zoom(3xcrop_mode) again to enable crop_rec and liveview should good now.

You can't use frame override , this is again known . The preset framerate is locked/fine tuned to that resolution size (besides why would what to change 23.976 ?)
You must expect the little feature sometimes will not work , What you can expect is to have a basic recordable extended resolution e.g. 2600x1200
Think of the 5d2 as a film camera , you have no auto functions  everything is manually set .  After all it is bleeding edge stuff .

My objective is to have a basic recordable extended res. in crop_rec that can be adjustable e.g. ISO & Shutter speed with
realtime liveview & HDMI support (at least at 480p) , which is useable now .

So I can conclude this preset is working then , I'm now adding more presets to crop_rec .
Of course that's easier said then done  :P 
I do have it figured out by the way just my C code skill are not the best (I'm slow)
But just like the tortoise and the hare I will finish the race , just not in record time  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on March 26, 2019, 09:27:12 PM
I also tested the new crop module - it works fine, the image is awesome. I was trying to see the picture via HDMI monitor for adjust the aspect ratio to see the borders of the image I shoot, but I couldn't... and all I see pink solid color with some kind of transparent image.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: honza123 on March 26, 2019, 09:54:49 PM
After a short test ...
Recording RAW 2624x1200pixels, 10 bit is OK
 :)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on March 26, 2019, 11:08:25 PM
I just was thinkin about crop factor reducing and thought that if we have money for buying sigma art 12-24 f4 or just Laowa 12mm f2.8 (which are zero distortion lenses) we can achieve 25mm in our crop module with 2.8 - 4 aperture, if we use sigma art 24mm 1.4 we get 50mm 1.4. Those lenses are costly, but have absolutely razor sharp and beautiful picture... So thanks Reddercity we have absolutely amazing 2624x1200 2:1 with full sensor bokeh style... Just wanna shoot cinema lol... The rest is to get a liveview (I would help you, Reddeer, but don't know how) and write a screenplay (also need to get some life experience for that, hehe...).
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 27, 2019, 11:47:04 AM
@Igor_Braun: you won't get full sensor bokeh style. You'll get a cropped bokeh - same "problem" as using a crop camera, because the used sensor area is cropped. 5568/2426=2.29. So your f/1.4 used in this crop mode will get a bokeh like f/3.2 on the full sensor.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on March 28, 2019, 05:32:30 AM
@dfort here a quick tutorial on DiGic Poke

from 5d2
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimage.ibb.co%2Fmuotkp%2FVRAM3_small.png&hash=dd2e1282587ae7a5db8e1d4a28c9fa43) (https://imgbb.com/)
Here a example of setting up Reg C0F06084 and of course this would be the same as c0f06088 (raw resolution)
The default valve for C0F06084 is 10036 which would mean 3x3 (FHD) & 30036 valve is for 5x zoom (3x crop_mode)

There Two ways to explore the poke , manually  setting valves  in the Valve box and see the results .
Second , set valve in valve box then with half shutter press in liveview , you can increase valve or decrease valve
This done with the Altering mode box , you can see the "x++" this increase the valve and you can change that to "x--"
to decrease , still with me ?

Now what can be adjusted with the poke , will all reg's to my knowledge but I could not see a way to adjust the cmos's reg's
so if you need a new offset e.g. vertical of horizontal not sure how to do that other then coded it in crop_rec.
You mainly need new offset's when you increase vertical resolution but not so much on horizontal .
When I coded 3008x1080 I didn't have to use any new cmos vertical offset's just timers & raw resolution reg's

So the important Reg's for basic extended raw resolution in D4 cams are
Code: [Select]
Timer"A" (C0F06008)
Timer"B" (C0F06014)
raw resolution (C0F06088)
A1ex gave me great explanation here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205614#msg205614) after that I understood how it worked  :)

This is for the 5D2 , I would think it's very close to 7D . I know on the 50D everything is the same except the CMOS reg's  where different .
Code: [Select]
              x1     x5 centered        photo    description
C0F06084:   0x10036     0x30036        0x10037   start row/column
C0F06088: 0x4F40432   0x46A04BA      0xEDD0B87   stop row/column (column: 1 unit = 2 pixels)
C0F06008: 0x23B023B   0x2770277      0x5DB05DB   FPS timer A (1 unit = 4 pixels horizontally)
C0F06014:     0x6D5       0x4FF          0xEDC   FPS timer B (1 unit = 1 pixel vertically)
CMOS[1] :     0xC00       0xE6A          0xC00   vertical start/stop
CMOS[2] :     0x40E       0x10E            0x8   horizontal pos & binning mode
CMOS[3] :       0x5         0x5            0x7   ???
CMOS[4] :     0x244       0x244          0x244   ???
CMOS[5] :       0x1        0x1           0xC05   ???
Not sure if you know how to read the  raw resolution reg (c0f06088)
let take the 5x valve for example 0x46a04ba , take the first half 46a= vertical  last half 4ba = horizontal .
The only other thing that you need to know is the head timers 3 & 4
Code: [Select]
c0f0713c
c0f07150

there control the vertical window in Liveview , so if you aren't increasing vertical res. then you don't really need to bother with it .
but it does help with reducing liveview overhead for increasing fps .

Still there ? I know there a lot to take in .
One last thing (I think  :D )
When trying to increase horizontal res. , you need to slow down timer"B" (c0f06008 )to around 9 fps first
then increase timer"A"(c0f06014) for extended resolution (1 unit = 4 pixel) be mindful to don't get in to the OB area by
reduce timer"a" too much .

I hope I didn't confuse you with tooooo much info  :D
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: dfort on March 29, 2019, 02:13:22 AM
Thanks!

Ok--here we go. My version looks a little different but it has all the necessary fields.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7898%2F46574056415_f6acc26e5a.jpg&hash=6292cab6d99d0c1da4a3b614e859c462) (https://flic.kr/p/2dXAdmF)

On the 7D the default value for that register is 100c3 so assuming it works like the 5D2 if it is changed to 300c3 it should be 5x zoom. However, if I put the camera in 5x mode that's not the value that shows up:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7859%2F40523392343_c5c37894a1.jpg&hash=3d6939fb949d4bad142b14b10a359f4e) (https://flic.kr/p/24JUYJk)

There Two ways to explore the poke , manually  setting valves  in the Valve box and see the results .
Second , set valve in valve box then with half shutter press in liveview...

How do you manually set the values? The only way it seems to work over here is using the half shutter press. I tried changing the value of C0F06084 and for the most part all I get is a garbled screen. I thought that maybe I needed to "Apply" option but that doesn't apply the value in the field, it applies a different value - it always changes to 10002.

...you can see the "x++" this increase the valve and you can change that to "x--"
to decrease , still with me ?


No, can't say that I'm still with you:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7834%2F47436486432_44e65dcd95.jpg&hash=991a7fedb05da132bddfb58103a8eb42) (https://flic.kr/p/2fgNoKJ)

So far I only figured out how to increment the value, can't seem to find the button to decrement.

Well, looks like this is going to be a bit more challenging. I'll check out that post from a1ex and see if anything sinks in.

Thanks for the lesson--I'll see if I can get any further with it a little later.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 29, 2019, 09:22:00 PM
How do you guys get the new crop_rec to work? Tried it with the build from first page. After enabling the neccessary modules I get a error message "tcc: error: undefined symbol 'lossless_decompress_raw' [E] failed to link modules". No RAW recording possible here.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 30, 2019, 05:01:48 AM
Sounds like you have MLV_Play enabled?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 30, 2019, 09:27:06 AM
Oh yes... thanks. This one was still enabled and I didn't recognize. Now it seems to work fine. The picture coming in look fantastic. On the right side I get some black lines. Is it as expected that the liveview has wrong framing? It is realtime as @reddeercity wrote, but recorded image has ~ double width and height to what I see in liveview.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: masc on March 30, 2019, 02:22:40 PM
Oh I love this resolution and the quality you get out of the old 5D2. Went out and did some test shots. Most of them look awesome. But some other shots are a bit strange and I don't know why yet. Some colums seems to have right blacklevel, others haven't. After around 50 to 200 frames all strips disappear.

First frame looks like that:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FdtfvZ35%2FM30-1246-frame-1.png&hash=d28c28c650fa5c33f4a6cd78ddfec4f8)
Frame 90 with same settings:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fp4XrmKr%2FM30-1246-frame-90.png&hash=88189a3fcf59a54b900b249f6f3acdce)

The black bar on the right should be easily cut somehow?!

And it seems that the frame is not exactly centered on the sensor - it is a bit more on the right, so the left part of the picture is a bit sharper than the right (right is a bit more to the lens corner).

Let me know if you need a shortend MLV for analysis.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: 70MM13 on March 30, 2019, 03:46:13 PM
that looks like agtg amplifier "settling" to me.
i see if often when shooting using the reduced gain mode i love so much.  it takes about 30 seconds to go away, and then it's stable for shooting.

maybe something is being set in the registers every time recording, instead of once only?

i'm just guessing, but it looks very similar to me...
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Galterius on March 30, 2019, 05:18:43 PM
Hey, I am new to the magic Lantern group !!
I own a 5d mark ii, and I decided to try 3k raw video on it.
All steps from the first page of this topic work, but there is one thing i don't understand


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FyY3Dbj1%2FFull-Size-Render.jpg&hash=ade134fe550a1858a10555b18d8b6445) (https://ibb.co/yY3Dbj1)


I think it is just because I am a little bit stupid, but what does that mean ?

Again sorry for my stupid question  :-X , and thanks for your amazing work... (who would have thought of 3k raw and a 10 year old camera...)

And also, is a 800x card enough to record 3k raw ? (and maybe 4k in the future) from magic lantern ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on March 30, 2019, 06:01:00 PM
The video modes discussed here (3K and others) are interlinked to liveview mode. Entering liveview and pressing magnify/loupe button (=called crop mode) enables recording in higher resolutions.

800x is for read mode only and has nothing to say for write performance. Lexar's 800x cards are doing very differently for different card sizes. 16-32 GB performs about 45 MByte/s, 64 GB does about 75 MByte/s. See www.cameramemoryspeed.com
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Galterius on March 30, 2019, 06:12:16 PM
thanks for your fast answer Walter Schulz, the button to press is the top right one, right ? and is it supposed to display something on the screen to see if its anbled or not ? I don't see a way to see if its on or off..., I might do something wrong, sorry  :-X

and I didn't know 800x was only for reading, thanks . If  I use a 120mb/s card, is it enough ?
Thanks again !
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Walter Schulz on March 30, 2019, 06:29:12 PM
Consult manual about how to operate your cam! You may have to press magnify button twice.

Rough calculation:
Horizontal resolution x vertical resolution x bit depth x frame rate x compression gain / 8 = Byte/s
4000 x 4000/2.35 x 14 bit x 24 x 0.5 / 8 = 136 MByte/s.
Cam might do about 75 MByte/s continuously due to limited storage interface.
https://rawcalculator.netlify.com/calculator_desktop
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Galterius on March 30, 2019, 06:34:30 PM
thanks i will consult my manual  ;) sorry for disturbing you  8)



It works !!!! Thanks a lot ^^?
 is it normal if I can't go to 2880 ? (I am stuck to 2560)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: jpegmasterjesse on March 31, 2019, 07:26:32 AM
It works !!!! Thanks a lot ^^?
 is it normal if I can't go to 2880 ? (I am stuck to 2560)

Use the build in the first post of the thread. Cheers!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: joanllm on April 01, 2019, 09:55:09 AM
Great work goes on like this! the quality of this camera's image is incredible! Just a question. Is it possible to increase the 1200 pixels in the future or is this really the limit? It's just out of curiosity.

Regards!

https://serving.photos.photobox.com/646633975cb85bff654605ffbada15d8acfd544241fa3c31b157ccba9d1f620fec945235.jpg

https://serving.photos.photobox.com/221122633a58bf77a484541c81752f7d1d8765f053386da7e7dc8fb680dcd6e0a707030b.jpg
https://serving.photos.photobox.com/05994318716afdbafebf049b4584fee8a200eff2a6cd83a20ee4138725ce64dc91c3e7c4.jpg

Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on April 02, 2019, 05:31:29 PM
Oh I love this resolution and the quality you get out of the old 5D2. Went out and did some test shots. Most of them look awesome. But some other shots are a bit strange and I don't know why yet. Some colums seems to have right blacklevel, others haven't. After around 50 to 200 frames all strips disappear.

First frame looks like that:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FdtfvZ35%2FM30-1246-frame-1.png&hash=d28c28c650fa5c33f4a6cd78ddfec4f8)
Frame 90 with same settings:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2Fp4XrmKr%2FM30-1246-frame-90.png&hash=88189a3fcf59a54b900b249f6f3acdce)

The black bar on the right should be easily cut somehow?!

And it seems that the frame is not exactly centered on the sensor - it is a bit more on the right, so the left part of the picture is a bit sharper than the right (right is a bit more to the lens corner).

Let me know if you need a shortend MLV for analysis.

I had the same problem sometimes but thought that the sensor's time is coming to its end. Hope I'm wrong.
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Galterius on April 02, 2019, 07:54:41 PM
thanks for your answers, I just got some sandisk 160mb/s card that should do the job ^^
My question is now, what is the best way, according to you, to import those .mlv files into Premiere Pro ? (to not loose the raw advantage too much, I don't really know what to do, there are lots of different technics on youtube but I don't know which one to pick  :)
Thanks !!!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: OlRivrRat on April 02, 2019, 09:33:36 PM
          @Galterius > Try Switch to Convert to Usable ~

https://www.magiclantern.fm/forum/index.php?topic=15108.0
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 03, 2019, 04:54:26 AM
Thanks @masc & @Igor_Braun for feed back , interesting never happen doing testing
but that what you guys are for  :P  I think know the issue . I'll post a fixed test build  a little tonight .

By the way what was the camera setting?
e.g. ISO , Liveview preview setting,  SR memory setting (e.g. 0-4) etc. ...
did you use any Auto setting? e.g. auto ISO or auto WB ?
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 03, 2019, 05:10:58 AM
......  Is it possible to increase the 1200 pixels in the future or is this really the limit? It's just out of curiosity.
Yes (1330) , at the cost of the reduced horizontal resolution -- manly because the sensor is too slow , main clock is 24 MHz
and reads 4 channels at a time so 24x4=96MP/s (mega pixels per second)
where the 5d3 has 8 channels at 24 MHz (24x8=192 MP/s). If I did the math right 2624x1200@ 24fps = 75MP/s so near the limit of
the image sensor 96 MP/s.
 
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 03, 2019, 05:58:52 AM
This off topic but I will answer it this time , better to post in the raw processing threads
...... what is the best way, according to you, to import those .mlv files into Premiere Pro ?
(to not loose the raw advantage too much, I don't really know what to do, there are lots of different technics on youtube but I don't know which one to pick  :)
Thanks !!!
Will it all depends on the computer hardware you have .
e.g. fast SSD , lot's of Ram 16-32GB & fast CPU ( at least a 8core i7 or AMD FX)
In all workflows , the best way is to use MLVFS (https://www.magiclantern.fm/forum/index.php?topic=13152.msg127218#msg127218) (Cdng on the fly , Virtual Cdng's) specially with 5D2
once you have mlvfs installed load MLV on to SSD ( does work will off the CF card but a little slow)
The big advantage is you don't have to export Cdng & use up hard drive space , just load them in to mlvfs start grading/editing no wait time .
Works in MAC , PC  & Linux , I've used it with Blackmagic Resolve & CS6 Adobe After Effects .
My preferred workflow on my PC (Windows) is Adobe After Effects (ACR) -> dynamic Link to Premiere Pro
Make a After Effects Composition , do your Camra Raw adjustment with ACR save it
In Adobe Premiere Pro, choose File > Adobe Dynamic Link > New After Effects Composition
You will need a very fast computer to do this .

My Mac workflow , is a little different depending on what I'm doing .
I all ways start with Adobe After Effect CS6 & MLVFS
If I'm doing heavy FX in Autodesk Smoke for MAC , I'll export either 12bit DPX or 16 bit Tiff image sequence
after I'm done I save as a ProRes 4444 XQ or 16 bit tiff sequence .
If I'm doing lite color grade & editing , I export 16bit tiff from AF+MLVFS to Apple compressor and save
it to ProRes4444 XQ+ alpha and finish off in FCPX ( it's truly my real favorite all around video App) IMO  ;)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 03, 2019, 08:47:00 AM
Updated 2.6k_5d2_4-3-2019_crop_rec.rar (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2.6k_5d2_4-3-2019_crop_rec.rar) , adjusted the right side offset .
So now it's 2592x1200 @ 23.976fps , the raw dump is 2618 but because of the "Mod 8" thing for resolution size it goes to 2592 .
Canon liveview (pinkish) blacklevel is off for some reason more then likely it's the liveview preview reg's are not right ,
I'll need to play around with them , for now just use Auto or ML gray scale preview .
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Galterius on April 03, 2019, 02:23:06 PM
thanks a a lot reddeercity and OlRivrRat.
My pc should handle it (I have a i5 6600k and 32gb of ram), I will try the new crop_rec in a few minutes !
and sorry for the offtopic, I won't do the mistake again !!
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: Igor_Braun on April 04, 2019, 02:27:55 AM
Thanks @masc & @Igor_Braun for feed back , interesting never happen doing testing
but that what you guys are for  :P  I think know the issue . I'll post a fixed test build  a little tonight .

By the way what was the camera setting?
e.g. ISO , Liveview preview setting,  SR memory setting (e.g. 0-4) etc. ...
did you use any Auto setting? e.g. auto ISO or auto WB ?

Where can I see "SR memory settings?"
I remember ISO was 1600 (I've tried with 800 also, and in my case the lower I go, the less likely this glitch happens), I didn't use any auto settings, WB was something like 5400... (actually, I don't remember exactly, but usually I use this WB when shooting in my room)
Title: Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
Post by: reddeercity on April 04, 2019, 04:17:24 AM
Where can I see "SR memory settings?"
Near the bottom of MLV Raw Video Tab or in the "mlv_rec.cfg" file
This is my mlv_rec.cfg file
Code: [Select]
# Config file for module mlv_rec (MLV_REC.MO)
mlv.video.enabled = 1
mlv.res.x = 7
mlv.bpp = 0
mlv.aspect_ratio = 17
mlv.write_speed = 6969
mlv.display_rec_info = 2
mlv.buffer_fill_method = 0  *** this is the SRM***

I remember ISO was 1600 (I've tried with 800 also, and in my case the lower I go, the less likely this glitch happens)
Ok that's what I'll looking for , High ISO seems to have problem with these experimental builds even the