3K/UHD 5D2 Raw development and Other Digic IV Cams

Started by reddeercity, April 06, 2017, 12:22:27 AM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

reddeercity

Ok --So I decided to forge ahead after urging from some forum members and help from a1ex.
So to get up to speed read the posts below to those members who are following the development , should help explain where I am in all this.
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186209#msg186209
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186213#msg186213
Basically I trying to get back to the 3584x1068 raw buffer I had working here
Stay tuned  :)

ItsMeLenny

Good to hear! I think many people are ready to step up if you need a hand.

nikki


Tullen

Quote from: reddeercity on June 26, 2017, 06:08:01 AM
Ok --So I decided to forge ahead after urging from some forum members and help from a1ex.
So to get up to speed read the posts below to those members who are following the development , should help explain where I am in all this.
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186209#msg186209
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186213#msg186213
Basically I trying to get back to the 3584x1068 raw buffer I had working here
Stay tuned  :)

Great to hear. As I said, I will be helping as much as I can from September onward.

scotophorus

Quote from: reddeercity on June 26, 2017, 06:08:01 AM
Ok --So I decided to forge ahead after urging from some forum members and help from a1ex.
So to get up to speed read the posts below to those members who are following the development , should help explain where I am in all this.
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186209#msg186209
http://www.magiclantern.fm/forum/index.php?topic=19386.msg186213#msg186213
Basically I trying to get back to the 3584x1068 raw buffer I had working here
Stay tuned  :)

This is great news. Big thanks to you, Alex, and all the other geniuses for pushing forward the development. I'm ready to test and help in anything you need.

reddeercity

Quote from: Tullen on June 27, 2017, 01:08:43 PM
Great to hear. As I said, I will be helping as much as I can from September onward.
Sounds good , In the mean time I would suggest familiarize yourself with this thread , 10-12bit thread and of course the  iso-research source code. (That's where I started from)
You will need a developing environment -- I use a VM (VirtualBox) on Window7 Pro with a preconfigured magic lantern clone of Ubuntu ready to compile ML . Download links for clone of Ubuntu  here
You can run the VM under Mac or Windows just get the correct version for your OS , if you feel adventurous you can setup your own environment @dfort has make some tutorial here and here
You need to be very comfortable with compiling , merging , update source with "HG" etc. ..... I work with a VM because it's faster & easier to get up and running in less then 30 mins .

 

whitelight

Hello, I just want to thank all the developers for the work done, I wish this will one day be ported to 7D, old camera but still doing an amazing job when used with ML.

flofifull

Quote from: whitelight on July 10, 2017, 12:53:18 AM
Hello, I just want to thank all the developers for the work done, I wish this will one day be ported to 7D, old camera but still doing an amazing job when used with ML.

+1

reddeercity

5d2 users --could I please get a screenshot of the edmac in video mode like in this post here I what to compare to my 5d2 , in fact it wouldn't hurt to see other digic iv cams also for reference , Thanks  :) 

Edit:
You can use the latest nightly , but for the 5d2 users could you do 2 screenshot , 1 with the latest nightly and the other with the adtg_gui build from my bitbucket download  here  please
screen shoot feature in the debug menu , you have 10's to get to the right window.

Tullen

Quote from: reddeercity on June 28, 2017, 06:41:46 AM
Sounds good , In the mean time I would suggest familiarize yourself with this thread , 10-12bit thread and of course the  iso-research source code. (That's where I started from)
You will need a developing environment -- I use a VM (VirtualBox) on Window7 Pro with a preconfigured magic lantern clone of Ubuntu ready to compile ML . Download links for clone of Ubuntu  here
You can run the VM under Mac or Windows just get the correct version for your OS , if you feel adventurous you can setup your own environment @dfort has make some tutorial here and here
You need to be very comfortable with compiling , merging , update source with "HG" etc. ..... I work with a VM because it's faster & easier to get up and running in less then 30 mins .

Thanks for the recommendations, hopefully will make me helpful a bit faster.

obiyan19

Quote from: reddeercity on July 17, 2017, 04:12:43 AMfor the 5d2 users could you do 2 screenshot , 1 with the latest nightly and the other with the adtg_gui

just because i read this tpic with interest, done with my 5d2 an 24-105 mounted on (& CF sandisk extreme 32 GB 120 mb/s udma 7)

the first image done with nightly




and the second with adtg_gui.



600D + 7D + 5D2  // 24-105 L is ~ 100-400 L is ~ 18-200 is ~ 18-55 isII ~ sigma 50 f1.4 EX

reddeercity

Thanks @obiyan19 for the screenshots
nightly_18_07_2017


adtg_gui_2017_Apr04


Screenshots from @honza123 , thanks
Nightly_2017_Jul03


adtg_gui_2017_Apr04


Good thanks , this lines up with my 5d2 --I wanted to see the buffers from different 5d2 cams and to make sure I didn't corrupt
my ram , all good now to processed to the next level .

ErwinH

Here is a screenshot of the 550D running Nightly.2017Jul16.550D109 unified.
 




Hope it helps.

reddeercity


70MM13

Hello, I joined just to post this encouragement for you to pursue this for 550D...
I'd LOVE to be able to use my 550D as a b-camera for filming 10 bit 24P raw 1080p!!!

Please be assured that your efforts will be appreciated and will keep many good cameras working for years to come!

I'm starting a new film this fall, and this would be excellent!

Please make it happen!
Thanks.

reddeercity

I can now replicate the 3.5k resolution on my 5D2 but it's extremely fragile (I suspect the buffer I'm using & or how it's coded for liveview is little unstable)


5D2 3.5k eXperimental
by RedDeerCityTV, on Flickr


LV-002_Liveview Raw Buffer Dump 5D2 3.5k eXperimental
by RedDeerCityTV, on Flickr


Screen Shot 5D2 3.5k eXperimental with ML B/W Preview
by RedDeerCityTV, on Flickr


5D2 3.5k eXperimental Memory Patches
by RedDeerCityTV, on Flickr



5D2 3.5k eXperimental Memory Patches Error
by RedDeerCityTV, on Flickr


5D2 3.5k eXperimental Free Memory
by RedDeerCityTV, on Flickr


5D2 3.5k eXperimental Edmac ScreenShot
by RedDeerCityTV, on Flickr

I uploaded my adtg.log file to my google drive for those who are following along , I still need to analyzes the information for more developing but feel free to download it .
https://drive.google.com/file/d/0BxNY4d4Aa7KuQUNtM2V4UnY1UEU/view?usp=sharing

Still lots to understand as everyone can see , I could not change resolution or aspect ratio seem to be lock to 3584x1068 .
That could be because of the preset from the 3K 5d3 1:1 in the adtg_gui.mo with ISO research branch --not sure thou.
So to recap I loaded adtg_gui.mo , crop_rec.mo , raw_rec.mo(not mlv_lite) on  change set bd8e89d9dccf+ (iso-research) that I complied locally on my VM
1st: load 1:1 3x crop_rec setting
2nd: with the adtg_gui.mo enabled I activated the Digic register then the 3k 5d3 1:1 preset
then adjust the image with the cmos 1 & 2 .

I modified  raw.c , edmac-memcpy.c
in raw.c I add this

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif

also in raw.c  changed to 0xC0F04500 from  0xC0F04508

#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04500
#endif

in edmac-memcpy.c change the r/w channel to 0xff & 0x0 from read 0x19 & write 0x03

#if defined(CONFIG_5D2) || defined(CONFIG_50D)
uint32_t edmac_read_chan = 0xff;
uint32_t edmac_write_chan = 0x0;


From the Liveview dump I can see that there is something there , seem to have the right size on the screen
Of course more reverse engineering needed , but there is some progress .  :)



nikfreak

May I just chime in and sum it up?


  • read this, this, check this etc. as well as all included links in that posts.
  • Initially Greg succeeded and got results by doing it like described here and a1ex replicated it successfully on Digic5 and first got it only working with freezing preview or grayscale.
  • I'm a little bit unsure but for me it looks like you might need to get edmac_raw_slurp first running/enabled on 5D2 (more see below raw.c) but a1ex might correct me on this
  • raw.c: Check this commit. It's used in conjuction with edmac_raw_slurp and is meanwhile part of unified branch with even updated code parts  8).
  • try it with adtg_gui like a1ex did by simply overriding/changing one or both registers C0F06084 / C0F06088 like Greg did initially. You probably simply need to activate zoom/crop and find an example value for 2.8k/3k or whatever and afterwards hope you get the overriden resolution been shown in mlv_rec...

So frozen preview or not. i would say 1st goal would be to replicate what Greg/a1ex did AND be carefully with whatever you are trying out above this post  :o
[size=8pt]70D.112 & 100D.101[/size]

reddeercity


tonij

Quote from: whitelight on July 10, 2017, 12:53:18 AM
Hello, I just want to thank all the developers for the work done, I wish this will one day be ported to 7D, old camera but still doing an amazing job when used with ML.

+2

I don't post and there are probably many others like me but I find myself following along discussions like these.
All I see is page after page of green matrix symbols and sit here in awe at the ones who are able to stop bullets mid flight  :)

dfort

@reddeercity

Just wanted to bump this because it looks like it was progressing nicely and suddenly stopped. Believe me I know how frustrating it is trying to get new features working. A while back I was trying to get lossless compression on the EOSM and 700D and had to put it aside. A while later ErwinH picked it up on the 700D and figured out the missing pieces. Following his lead I was able to get the EOSM working.

What made this possible was sharing the code on bitbucket. Checking up on your bitbucket account I see that you forked the ML repository a few times, the last activity was back in March. A lot has happened since then. If you could keep your development current in bitbucket maybe some of us could offer suggestions.

There are several good articles for working with bitbucket and Mercurial (hg) and if you have any questions there are ML forum members willing to help out.

ilia3101

Quote from: dfort on August 23, 2017, 03:16:31 PMIf you could keep your development current in bitbucket maybe some of us could offer suggestions.
+1
Please do that, I'm wanting to test out what you're doing, but the verbal 'patches' are just too cumbersome to keep up with.

Also what exactly is a 'liveview buffer'? How is it different from just a piece of memory?

tecgen

+1

Could you please share the source code? Maybe you could simply push all your changes as one commit. We can compare the sources against the master/trunk within BitBucket afterwards.
Canon 5D Mark II, 50D, 550D/Rebel T2i, EF 40mm f/2.8 STM, Sigma 18-35mm f/1.8, EF 85 f1.8, EF 135 f2.8 SF, Zoom H2n

reddeercity

Sorry guys I haven't had time to responded to the posts , had one of my main servers crashed (lost the raid & the backup  :( ) so I had to rebuild from scratch which took some time.
@dfort ,Ilia3101, tecgen
I really don't know how share my code on bitbucket , I work locally in my VM of Ubuntu which is very comfortable for me to develop in.
I tried once to set it up but could not figure it out , so I just kept working locally .
I did post my source code in my Bitbucket downloads , but all my changes are very simple I posted all my code changes and I thought I explained it in other posts.
I know this is not the way every one works , but for me it's the easiest way , so unless some one can tell me in very simple terms how you use/share my code on bitbucket correctly
with out  pointing to badly written tutorial I would gladly comply. This may slow down development , until then sorry I'll have to keep what I'm doing .

I'll try to explain what I have done and why . Here the Source Code I been working from , it's has not been modified with my code .

I guess I should post a declaimer first -- I'm not a full time coder/hacker I know how thing work , I read code and understand perfectly just don't know the proper syntax when writing c code
I have to see example and do a lot of reading to make sure I'm doing the syntax correctly.


For this to work there needs to be a customs raw lv buffer , other then the canon which is for the default 1:1 (1880x1250)
So I followed a tip  from a1ex about the 1100D in the 4K branch and I use the largest buffer I could find in the Edmac debug menu
I use the 3x crop mode buffer because it was big , I use this from Greg to help get the buffer but it unstable , could be because I did not disable the Canon default liveview buffer when I tested my code
In RAW.C  I added the new LV buffer
#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif

With out the buffer it will not work
Quote from: a1ex on April 18, 2017, 11:55:34 PM
The custom buffer is required, as ML redirects it only while recording.
In the crop_rec_4k branch I've used a SRM buffer (which can accommodate a full-res 14-bit picture).
This needs to be passed to SetEDmac as the first argument (instead of Canon's default buffer).
The buffer redirection without CONFIG_EDMAC_RAW_SLURP is fragile: it relies on lucky timing. It's best refactored somehow, but cache patching is also ugly...
Also in RAW.C I change the memory addressed for Liveveiw default was 0xC0f04508 to 0xC0f04500
Reason was it's the Photo Liveview buffer address and I thought logically it should have what I need to get to 3.5k and I was right to a certain degree .

#if defined(CONFIG_5D2) || defined(CONFIG_50D)
#define RAW_LV_EDMAC 0xC0F04500
#endif

In edmac-memcpy.c I also changed the default R/W channels , default Read 0x19 & Write 0x03
Reason , it seem to been the r/w for the larger buffer I'm using , but this my not be the case as I could can not save a dng in the liveview buffer dump just the YUV 422 dumps
as per post #90 http://www.magiclantern.fm/forum/index.php?topic=19336.msg187766#msg187766
#if defined(CONFIG_5D2) || defined(CONFIG_50D)
uint32_t edmac_read_chan = 0xff;
uint32_t edmac_write_chan = 0x0;


That's It for code modifications the rest is done with  adtg_gui.mo with ISO research branch , by enabling 5D3 1:1 3k preset and adjusting the cmos registers .
See post #90 for details , not sure if there a better way to implement the custom buffer "RAW_LV_BUFFER_ALLOC_SIZE " this need more work to understand
I would say this would be the first thing to focus on -- that about it for now if you or any one have question about what I have done just ask I do my best to answer .

ilia3101

Quote from: reddeercity on September 03, 2017, 02:06:21 AMSorry guys I haven't had time to responded to the posts , had one of my main servers crashed (lost the raid & the backup  :( ) so I had to rebuild from scratch which took some time.
Oh no :(  Hope you didn't lose any important footage.

Quote from: reddeercity on September 03, 2017, 02:06:21 AMI know how thing work , I read code and understand perfectly just don't know the proper syntax when writing c code
Exact opposite here, know C, but have no idea how Magic Lantern or Canon works :D

Quote from: reddeercity on September 03, 2017, 02:06:21 AMI really don't know how share my code on bitbucket
Ok, well even in that case, you can still just upload a zip file with your modified code.
I think that is still better than describing what code you added to what files, as I don't know what part/line of the file you add stuff to, as that could matter in some cases, if you uploaded it I wouldn't have to worry about that as much.

But thanks, I will have another go and see if the same thing works on my camera tomorrow.

When you have the ~3500x1000px buffer showing in the menu, is the camera in a state where it can record a small test clip at that resolution?
Or would it just be noise and garbage? If so, what would the steps to getting a clean image be from that point onwards?

Also what do developers like a1ex think of reddeercity's work?

reddeercity

@ Ilia3101 here you go Source Code with modification magic-lantern-adtg-gui-4k-test_source_code.zip
In  edmac-memcpy.c Line 16 & 17 modified
In raw.c Line 78 , 116-118
in raw_rec.mo line 94 & 95 add resolution frame size 3840 & 4096 .
"Note" this is only working with raw_rec.mo not mlv_rec.mo
Quote from: Ilia3101 on September 03, 2017, 11:50:10 PM
Exact opposite here, know C, but have no idea how Magic Lantern or Canon works :D
That good to know , that will help a lot  :D

Quote from: Ilia3101 on September 03, 2017, 11:50:10 PM
When you have the ~3500x1000px buffer showing in the menu, is the camera in a state where it can record a small test clip at that resolution?
Or would it just be noise and garbage? If so, what would the steps to getting a clean image be from that point onwards?
Can't save a dng yet , tried a image dump I can save the liveview422 see  post #90  here is the original LV dump 422 image  LV-002.422 in Black & White ML perview  I guess I should correct my self it did save a DNG file but it was "0" bytes and there was a HD Liveview dump image HD-002.422 which was normal looking .
I think to clean it up it could be as easy as fps A & B timing  or even the  Cmos  register , once there a clean LV image with ML B/W preview I think it may save a dng dump , there a raw record error when I try to save .raw file (Yes I'm still using raw 1st version not mlv lite , it's very basic just record raw once I have something stable I'll move to MLV Lite)

Quote from: Ilia3101 on September 03, 2017, 11:50:10 PM
Also what do developers like a1ex think of reddeercity's work?
http://www.magiclantern.fm/forum/index.php?topic=19336.msg187776#msg187776
http://www.magiclantern.fm/forum/index.php?topic=19336.msg184661#msg184661
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183023#msg183023
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183200#msg183200
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183374#msg183374
http://www.magiclantern.fm/forum/index.php?topic=19336.msg183461#msg183461
I guess I could go on & on , if you are asking is this a realistic development or even possible  , my belief is it is possible.
Or else I don't think I would have got so many tips from a1ex and nikfreak and other developer .

@ Ilia3101 I will also PM you with the test 3.5k build I was using to do my test  that I posted in #90 , I don't what to post it publicly so it get out in the wild.