3K/UHD 5D2 Raw development and Other Digic IV Cams

[SoulState]

And this is without 3x desqueezing:

[theBilalFakhouri]

I want to ask - is this normal in 3x1 mode? (50d) Looks like terrible artifacts...

It's not normal, it's coming becuase of bad configuration . .
The dev of this thread wasn't active for years.

[Milk and Coffee]

Hi all, been getting occasional pink frames when filming with an HDMI monitor plugged in. Any ideas?

[Lis_Rudy_PLR]

hello, i got eos 1200d and i really want go fhd48fps or even 45fps. The problem is that i actually complete nooby in programming etc. Could please someone help me to somehow port crop_rec to this cam? I was looking for it everywhere on forum but there's almost nothing for this camera bout cropping etc. im sending screenshot with error that shows when trying to load crop_rec

[Walter Schulz]

Mission impossible.
1200D's SD-card interface is locked at 25 MByte/s and ATM there is no way to tweak it.

Rough calculation (with all the bells and whistles not available for Digic 4):

1920 x 1080 x 45 fps x 10 bit x 0.4 (compression) / 8 = 46.7 MByte/s

Far beyond technical specs for continuous recording.

[Lis_Rudy_PLR]

oh, so that's how it is. so basically the last thing for me to do is wait for someone to break the card write limit on 1200d somehow or just get cash for newer camera. thanks for reply anyway.
Also the continous recording is something that i don't need, if the video recording would be stable for around 2 minutes or so it would be great (it is like this with 1.7x bitrate and it's enough for me)

[Walter Schulz]

We are talking about different things.
Bitrate manipulation is for native H.264/MOV recording only and doesn't apply to raw/mlv recording.
Required bandwidth without tweaking is around 5.5 MByte/s. 1.7x should give around 9.4 MByte/s and that's within cam's limits.

[reddeercity]

After seeing considerable work on D6 (7d mrkii heder)  &  D7& D8(names_are_hard)   
Its inspired me to finish some unfinished magic lantern business on the 5D2 & 50D
I find that i have some free now (6-8 weeks , after that Drag racing Season(ProMod's) start around Mid May (canada)
being i build race car chassis & parts (I'm in the metal fabricator business) i need to be at the track & late nights in the shop 

Anyway i'm going to take a stab at CF card over clocking first, I have all the code (thanks a1ex!) and it work on the old
ml core platform (.raw video) optimize for D4 cams with no restrictions on performance , That not why I'm using the old core ,
mainly because it work or should i say it will once i can compile it , then i can that the info and implement it in to the newer core that
D4 works on crop_rec, if i can get the CF card controller to switch to UDMA 7 from UDMA 6 i should go from 80-85MB/s to 120MB/s or around there.
But here lay the problem , see screen shot , how do i add "gcc 4.8.3 or check for it , i'm on Ubuntu Bionic 18.04
I do all my crop_rec development with it so no issue there just this old core stuff ,
do know for a fact that a1ex had it working on the 5D2 in the very early days of ML


Any help or advice would appreciated , i'm not big Linux user mainly PC(windows7 some windows 10 if i have to)
and some mac. its been about 3 years since i did any development so just need some refreshing 

A few other things i what to finish/complete on 5D2 & 50D
Compressed 14bit raw (I now know what the problem is "ResLock address issue" 5d2 would lock up on saving the compressed raw so
I've already had it running serval years ago but drop it because i couldn't find the issue .
a fully funtional 1x3 crop_rec perset without adtg_gui module, the problem is the adtg reg are not applying for pixel binning "more then likely just a coding problem" I'm not a professorial coder , but i can get by.
Better Preview in crop_rec & general code cleanup (sometimes there a race condition in crop_rec between 3x1 4k (4096x590->4096x1770 2.3AR)
and High speed frame rate 1x1 48FPS 1880x774 .
Just for the 50D :
Get the ADTG Reg to apply , seems to be a problem on D4's ? or my code .
In crop_rec on 50D i can get 3760x1080 , the problem is the vertical , I need to have an ADTG reg applied to get a full vertical
(5d2 dosen't have this more then likely because it all ready does video out of the box)
the 50d never had Video even thou it was always there , Magic Lantern code enabled it so crop_rec (5xzoom) was always limited to 1080 vertical
until i found the Reg (only but luck) so i do 1x3 on 50d but i need to fix the ADTG reg first then everything will fall in place fast on the 50d.

That all for now , i hope all development continues on the forum for everyone to see publicly , i don't like when the ML discussion is behind closed doors
that one of the reason i stop doing any development , even thou the development is on D6 ,7 & 8 it still helps me a lot discover thing on the D4 cams.

[Walter Schulz]

Good to have you back on tracks. Can't help with your dev environment issue. The only thing I know (IIRC) Ubuntu is not the prefered dev platform (to put it into friendly words).

That all for now , i hope all development continues on the forum for everyone to see publicly , i don't like when the ML discussion is behind closed doors
that one of the reason i stop doing any development , even thou the development is on D6 ,7 & 8 it still helps me a lot discover thing on the D4 cams.

Not getting the "closed door" bit. Most of dev talk today takes place on our Discord server. Only essential findings will get documented in forum which serves as some kind of "long term memory". I think apart from Danne all current devs and contributors are present over there. You will find a discord invite on top of this page: https://discord.gg/uaY8akC

There is a restricted area but it is for administration.

[heder]

After seeing considerable work on D6 (7d mrkii heder)  &  D7& D8(names_are_hard)   
Its inspired me to finish some unfinished magic lantern business on the 5D2 & 50D

Yes, come back and help lighting up the development 

see screen shot , how do i add "gcc 4.8.3 or check for it ,

• Download the arm gcc packages https://launchpad.net/gcc-arm-embedded/4.8/4.8-2014-q3-update
• Unzip it somewhere
• Add the path to Makefile.user (if needed update "ARM_ABI", "GCC_VERSION" & "PREFERRED_PATH")

[names_are_hard]

After seeing considerable work on D6 (7d mrkii heder)  &  D7& D8(names_are_hard)   
Its inspired me to finish some unfinished magic lantern business on the 5D2 & 50D
I find that i have some free now (6-8 weeks , after that Drag racing Season(ProMod's) start around Mid May (canada)
being i build race car chassis & parts (I'm in the metal fabricator business) i need to be at the track & late nights in the shop 

Cool, would be good to have you back

That all for now , i hope all development continues on the forum for everyone to see publicly , i don't like when the ML discussion is behind closed doors

Keeping devs involved is the most important thing for me, so I'd like to understand this more.  I'm not aware of anything being closed up, apart from the way Alex used to work.  Is there anything in particular that you don't like?  Maybe we can imake things better.

I would say the dev work has been more public since Alex left than before.  Alex did a lot of work, but he didn't like sharing things until they were finished.  We had to relearn lots of things about D678 cams because it wasn't documented anywhere.  E.g. the M50 "fishy" build - the source for this was never published, or even shared with other devs.  This was a largely working Digic 8 build, I could have learned a lot from it.  Instead I had to work it all out myself.

See updates from me since at least 2020, here:
https://www.magiclantern.fm/forum/index.php?topic=19737.350
Kitor on edmac, here:
https://www.magiclantern.fm/forum/index.php?topic=26249.0

Summaries of work here and here:
https://www.magiclantern.fm/forum/index.php?topic=26814.0
https://www.magiclantern.fm/forum/index.php?topic=26852.0

Also see a large number of commits over the last few years, all open   https://github.com/reticulatedpines/magiclantern_simplified/commits

Discord is just as open as the forums, anyone can join.  If I posted all my small findings to the forums it would be useless spam.  Instead, I post summaries, with links to interesting commits.  We treat Discord as useful for quick discussion, but useless as an archive, so we document things properly on forum and in the repo.

PS I did lots of work on D6 too   Heder started from a working ML build for 7D2, because I did that port.  750D and 5D4, too.  There's been lots of background work getting ML booting on many new cams!

[names_are_hard]

Separating out code / repo stuff.

Any help or advice would appreciated , i'm not big Linux user mainly PC(windows7 some windows 10 if i have to)
and some mac. its been about 3 years since i did any development so just need some refreshing 

The repo I maintain should work on old cams, and has many branches merged.  It contains crop_rec, lua_fix and other branches: https://github.com/reticulatedpines/magiclantern_simplified/commits

I mention this because some of the work I've done means this repo is much easier to build on modern systems.  You don't need to find gcc 4.8.  You can use any gcc from 8 to 13.  Ubuntu is an okay distro to use, Debian is better (this makes some things easier with qemu testing).  It compiles cleanly, with no warnings and no errors.  If any compilation warnings are introduced, I fix them.

Unlike the heptapod repo, I want to keep the number of active branches very low: only "dev".  All other branches are temporary, with the intention they will be deleted or merged into dev.  I found using heptapod very confusing - hard to find what you wanted.  Hard to know if a given branch would build, or what features it had.

I've also done similar work with qemu code, making this easier to build, and easier to run.  No patching things, no needing weird old versions of tools.

If you want to try this repo, I will happily talk you through how to get it working.  It is likely to become the official repo.  Nobody with access to the current official repo is active anymore, and making a working dev environment with it is difficult.

[reddeercity]

Yes, come back and help lighting up the development 

I do my best 
I'm in catch up & refresh mode , being a while starting to get my head around everything .

• Download the arm gcc packages https://launchpad.net/gcc-arm-embedded/4.8/4.8-2014-q3-update
• Unzip it somewhere
• Add the path to Makefile.user (if needed update "ARM_ABI", "GCC_VERSION" & "PREFERRED_PATH")

Doing this right now , thanks its coming back to me now .

[reddeercity]

@names_are_hard , thanks , glad to be back .
I guess Discord just sound unappealing , having to join a different forum/chat , as I'm not very good at chatting/typing fast/spelling 
Take me forever to type up these posts on magic lantern (spell checker etc. ...)
I have dyslexia so it's a challenge but funny thing about It I have no problem with reading & write code (go figure) .

If I posted all my small findings to the forums it would be useless spam. ...... We treat Discord as useful for quick discussion, but useless as an archive, so we document things properly on forum and in the repo.

Ok I don't know that , I've always found all the info needed here on the forum for the stuff I did in crop_rec , Actually a lot of information came from
[CMOS/ADTG/Digic register investigation on ISO] and others related.

I've being reading all of your posts of late & have tried to keep up with the 200d from you(great work) .
I actually was very interested in the 5D4 (D6) a few years back but the cost was just too much , even today these still around $2,000.0 (Canadian) used.

So 2 Thing that really brought back to do more work was 1st: Heder 7DMrk2 work simple great work at getting access to the raw stream & the different ways he achieved it plus
the quick access to the system Then he summarized on how it was done . Besides that the 7D2 Mrk2 is a very good cam 3x3 liveview no line skipping , clean HDMI with audio & 3K 5x zoom
I would say it may be better then the 5D3  specially it raw vide is fully functional .Oh yea it a lot cheaper too , I can pick one up used from camera rental store for $599.00 (Canadian).

The second thing that got me back was when to posted in the ResLock thread https://www.magiclantern.fm/forum/index.php?topic=6740.msg245655#msg245655
got me thinking about 5D2 address's,  so I was Hooked again! I have a need to finish this plus a few more thing I would like get working if possible
E.G. MJpeg 4.2.2 (color space) (AVI) compression (same as 1DC)
Change the Uncompressed liveview stream from 4.2.2 to 4.4.4 also if its in the Jpeg encoder chip change liveview from 8bit to 10bit .
I do know the HDMI has 3 different color space out put , ok 4 their a 4th "Unknow color mode" maybe Linear color similar to EXR's or DPX's files or the Raw color space .
Something to need to used a 2.20 lut curve to see the image etc. .....
here a short summary out of the 5D2 rom disassemble , just did a "HDMI" search with visual Studio on win7

Code Select Expand

475223: "Pana_TurnOnHDMI (Audio = %d)":
   475247: "Pana_TurnOnHDMI : EDID_COLOR_444":
   475257: "Pana_TurnOnHDMI : EDID_COLOR_RGB":
   475267: "Pana_TurnOnHDMI : EDID_COLOR_422":
   475277: "Pana_TurnOnHDMI : Unknown ColorMode (%d)":
   475289: "Pana_TurnOnHDMI : FULL_HD_50":
   ....................
   475417: "Pana_TurnOnHDMI : FULL_HD_60":
   475426: "Pana_TurnOnHDMI : DEFAULT_NTSC":
   475435: "Pana_TurnOnHDMI : DEFAULT_PAL":
   475444: "Pana_TurnOnHDMI : Unknown VideoCode (%d)":
   475457: "Pana_TurnOnHDMI : Av_hdmi_avreq_init failed":
   475469: "Pana_TurnOnHDMI : End":

I would like to get in to the Jpeg chip , on the 5D2 its the JP62 if I'm not mistaken , there so much I thing in there that would help us .
I could go on and on , time to get back to it 

[names_are_hard]

If you never used IRC (or never knew that ML devs used IRC!), then you definitely don't need to care about Discord.  It is better for quick things, so feel free to use it if you just want to ask a question and get an answer (it's like WhatsApp or similar in this regard).  Can also do screensharing / voice chat, sometimes useful.

I like the 7D2.  Good build quality, metal body, even has weather-sealing.  It feels like an APS-C sized 5D3.  It's faster with card spanning than 5D3.  I'd recommend it if you see one at a good price

5D2 plans also make sense to investigate.  I'd especially appreciate somebody with more experience working with the hardware, who could try to verify my theory that the reslock structures are a linked list holding (block num, device ID) pairs.  If I'm right, it means the way ML locks devices is wrong.  So if some cams don't work the way they "should", that could be the cause.  And the read_edmacs[] / write_edmacs[] can be simplified quite a bit.  We might also be able to determine what individual devices are for - the current way we do this means we're using the wrong numbers!

Interesting HDMI strings   Good luck!

[SKecorius]

This is fantastic news - welcome back, reddeercity!
Must admit that I bought 5D mark ii only because of ML and your developments. Was always keen on trying some extended features, with less interest in actual scenes to capture  .
After some months of silence in 3K/UHD 5D2 thread, also Zeek disappearing from youtube, I was about to let my camera go...
So you can imagine how happy I am now

Looking forward to exciting developments and news from you!

[reddeercity]

Making some progress i guess , got it to compile . Took the cf_acc module code and put it in my new crop_rec source that seem to work until it didn't .
Got a error i had years ago when i was working it activity , didn't understand the problem then but i think I understand now but don't know how to solve.
Here what i got so for see image below:



so its looks for

Code Select Expand

<linux/types.h>
and there's nothing in the magic lantern source , then i looked in the "Makefile" in the module and found this

Code Select Expand

# define the module name - make sure name is max 8 characters
MODULE_NAME=cf_acc
MODULE_OBJS=cf_acc.o hdparm/identify.o hdparm/hdparm-routines.o hdparm-ml-shim.o

# include modules environment
include ../Makefile.modules

LINUX_PATH=$(HOME)/src/linux-3.19

CFLAGS += -Ihdparm
CFLAGS += -Idietlibc/include
CFLAGS += -I$(LINUX_PATH)/arch/arm/include/uapi
CFLAGS += -I$(LINUX_PATH)/arch/arm/include
CFLAGS += -I$(LINUX_PATH)/include/uapi
CFLAGS += -I$(LINUX_PATH)/include

# fixme: properly handle generic-y from Linux makefiles
# (hack: copy and rename folders around manually)
CFLAGS += -I$(LINUX_PATH)/generic

test: FORCE
gcc hdparm-test.c hdparm-ml-shim.c hdparm/identify.c hdparm/hdparm-routines.c -Ihdparm -o hdparm-test
./hdparm-test

So it needed something from Linux or it needs the uapi from arms ? i don't understand this never seen this before , something a1ex needed to run
the overclocking on the cf card . i hope somebody could enlighten my please , I'll do more research in the mean time.

[reddeercity]

Did a search for all references to "linux" in the "cf_acc module to see what its calling etc. ....

Code Select Expand

hdparm-ml-shim.c:
  7: #include <linux/types.h>

Makefile:
   9: LINUX_PATH=$(HOME)/src/linux-3.19
  13: CFLAGS += -I$(LINUX_PATH)/arch/arm/include/uapi
  14: CFLAGS += -I$(LINUX_PATH)/arch/arm/include
  15: CFLAGS += -I$(LINUX_PATH)/include/uapi
  16: CFLAGS += -I$(LINUX_PATH)/include
  18: # fixme: properly handle generic-y from Linux makefiles
  20: CFLAGS += -I$(LINUX_PATH)/generic

hdparm\hdparm-routines.c:
  6: #include <linux/types.h>

hdparm\hdparm.c:
    27: #include <linux/types.h>
    28: #include <linux/fs.h>
    29: #include <linux/major.h>
   825: * The Linux kernel IDE driver (until at least 2.6.12) segfaults on the first
   830: * We would like to issue these commands consecutively, but since the Linux
  1632: " Due to bugs in older Linux kernels, use of these commands may even\n"

hdparm\hdparm.h:
  91:  * instead of using the system def from <linux/fs.h>

hdparm\identify.c:
  7: #include <linux/types.h>

hdparm\sgio.h:
  3: #include <linux/types.h>
Or should just comment out all these references ?

[reddeercity]

Ok I thing i figured this out , i need to put the Linux parts in my "Path" for compiling .
Been learning all about the "Linux Path" environment for developing, etc. .....
Finally found a list of command for terminal command line , it's really not that different then old DOS or 4DOS 
It all coming back now  , had to deep dive in to the "Path" stuff now i can move on and make some progress .

[names_are_hard]

Sorry, I missed some of this as I was busy with digic 7 raw video.

You almost certainly don't want <linux/types.h>.  That's typically used for writing linux device drivers.  And we want ML to build on Mac and Windows, too.  You probably want <stdint.h>.  Swap that over and see if anything breaks.

You definitely don't want "LINUX_PATH=$(HOME)/src/linux-3.19".  That's just wrong.  Can you link to the source for this module?

[reddeercity]

Sorry, I missed some of this as I was busy with digic 7 raw video.

No worries , raw video on D7 that great !! 

You definitely don't want "LINUX_PATH=$(HOME)/src/linux-3.19".  That's just wrong.  Can you link to the source for this module?

Ok , i found the source I'm using in foss.heptapod.net
Here's the link to the module , it looks the same as my source i downloaded about 4 years ago
cf_acc hdparm

[reddeercity]

@ names_are_hard
Here other post that references cf card sppeds & cf_acc hdparm
CF Card Info screen shot https://www.magiclantern.fm/forum/index.php?topic=19336.msg207054#msg207054 ,
a1ex links the "hd_parm" build https://www.magiclantern.fm/forum/index.php?topic=19336.msg207053#msg207053 ,
direct reg for UDMA7? https://www.magiclantern.fm/forum/index.php?topic=19336.msg207128#msg207128 ,
some useful info , its where i got the idea to over clock the CF card bus from a1ex https://www.magiclantern.fm/forum/index.php?topic=12862.msg199697#msg199697
From a1ex on cf speed & overclocking

I was doing sector-level benchmarks with Canon routines from bootflags.c, filling the entire card with 0xFFFFFFFF, 0x00000000, 0x55555555, 0xAAAAAAAA, 0x5A5A5A5A etc,
to see if that makes any difference in benchmarks. It didn't, but some blocks were written at ~120MB/s, others at ~85MB/s, and a very small percentage of them were written at lower speeds.
That pattern wasn't exactly repeatable, but the histograms were pretty much the same .....

@reddeercity: more details after I'll get a new card (it *is* possible to overclock the 5D2 CF interface).

https://www.magiclantern.fm/forum/index.php?topic=12862.msg199759#msg199759 a1ex quoted"You can send ATA commands to the CF card (QEMU emulates them), so if the only difference between UDMA 6 and 7 is timing, it might even be possible to put the card in UDMA7."

[Walter Schulz]

I'm not sure if my information is correct but UDMA-7 may come with a lower voltage. No CF 6.0 spec available here, though.
Maybe ted ramasola can help. He seems to be (or was) on good terms with Komputerbay.

EDIT: Found a piece of contradicting info claiming UDMA-5, 6, 7 all using 3v3.

[names_are_hard]

Here other post that references cf card sppeds & cf_acc hdparm

Thanks, I understand what's going on now.  This module is a port of the existing linux hdparm util so we can run it on cam.  How successfully, or how to even interface with it, I didn't look into   I assume the intent was to learn what control was possible over CF cards, then make nicer (and importantly, smaller) code to do the parts we care about.

The code is obviously quickly hacked up, possibly why it's in cf_acc module (which already existed).  That combined with it being a low-level linux util explains the refs to linux/types.h.

I've removed the linux specific stuff, here: https://github.com/reticulatedpines/magiclantern_simplified/tree/hdparm_hack

That should build more easily.  Because it's based on my repo, you can use a more modern linux.  I recommend Debian Testing.  It might not work on whatever old linux you're using, mainly because we converted everything to python3.  Try it and see if you want.

I haven't tested the code in any way.  Will it destroy your cam?  Maybe!

[reddeercity]

Found a piece of contradicting info claiming UDMA-5, 6, 7 all using 3v3.

I found in the documents that the CF cards  are 5v tolerate 
https://www.transcend-info.com/support/product/memory-card/compactflash-1000#SpecificationsModal
says there cards are 3.3v & 5V , i think the 5v is from the older ide PC interface like cd roms & 3.5' floppy disc etc. ....
when CF cards where used for backup soooo long ago