3K/UHD 5D2 Raw development and Other Digic IV Cams

[reddeercity]

I'm taking a break from magic lantern , so don't expect any update .

[IDA_ML]

I am really sorry to hear that, Reddeercity.  Now, that you achieved such a remarkable progress with the 5D2, provided so much useful information for all D4 cameras and there is finally hope for 4K-crop recording on them.  I understand, you feel humiliated but please be aware that your efforts are greatly appreciated by many people on the forum.  That is why, I hope, you will reconsider your decision and will continue providing updates on your work. 

[12georgiadis]

@Reddeercity, you helped us a lot with your research. Please don't give up. You've done an amazing work for D4 improvement.

[masc]

@reddeercity: sad to read that. I enjoyed every day reading your messages, what you found out. You are so close now - the goal should not be far!

[ilia3101]

@reddeercity Thank you for all you have done! I can imagine you are tired, even from the occasional checking I could see it was a lot you were doing. Really excited to get back to this stuff when I get my camera back

[reddeercity]

with the value of  1000102,  I have get a big improvement of around 30% speed write.
Sadly , this corrupt the card filesystem .  ( i can't read mlv file in card and i must format the card if i try to delete it)
........hey , reddeercity, do you hear this? 

Yes , nice -- Can you do a benchmark in cam with out it corrupting the file system ?
would be nice to see a LOG file . either in dm-spy or mmio to see what the card connects at .
I'm wondering if there more to UDMA7 then the reg. e.g. CF Timing

Code: [Select]

83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6So I would expect to see

Code: [Select]

DatMod = 7assuming this is UMDA but not too sure what to see for

Code: [Select]

DatTim = 3maybe ?

Code: [Select]

DatTim = 4A LOG file would confirm this
Or does the cam go to auto mode went UDMA7 is forced
This is from dm-spy log reviewing a cr2 or a h264 .mov file , not sure which one .

Code: [Select]

825FE>  CSMgrTask:00095f98:00:00: *** register_interrupt("CFDriver", 0x82, 0xffb8b8cc, 0x0), from ffb8bb58
834A0>  CSMgrTask:ffb8a92c:22:05: [ID:Model Number] = LEXAR ATA FLASH CARD                   
834C8>  CSMgrTask:ffb8aabc:22:05: IDE = 4, PCMCIA = 80, UDMA = 6
83572>  CSMgrTask:ffbdb8a0:22:01: cfDecideTiming: UDMA Mode 6 (CFA4.0)
83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6


To all the core supporters  , THANKS
I'll keep checking in .
Later I'll post a explanation for my decision to pull back from ML 

[a1ex]

The hardware only cares about what gets written to MMIO registers (or DMA channels). What hardware we've got here?
- CF controller (from camera); likely that's where the data transfer clocks are generated
- CF card itself (this one may also have to know about UDMA mode; it's configured with IDE commands, also sent via MMIO)

What gets into debug messages (DatTim, DatMod, UDMA Mode) are just some variables in Canon code that only live in the ARM processor, i.e. they don't reach the hardware. Sure, their value decides what reaches the hardware, but in many cases we can overwrite that later (and that will happen without changing the original debug messages). The hardware doesn't look at debug messages - these are just human-readable messages written by Canon engineers for themselves.

So, you won't necessarily see DatTim = 4 or DatMod = 7, unless you patch Canon code to print that. Either you change that message (easy, but the hardware won't listen), or you change the message AND add the code to handle these values (not present in the original Canon firmware), or - sometimes easier - you just write a different value to these registers later. That last approach (i.e. changes done via DIGIC Poke) won't show up in a regular DebugMsg log, but will show up in a MMIO log.

So, the first half of the story is probably covered by the registers waza tried. We know they are somehow related to clock generation, lower numbers appear to result in higher frequencies, but that's pretty much it.

The second half of the story may or may be necessary. Some cards are able to work at (slightly) different speeds than what was declared in the IDE commands, others maybe not.

Sorry I'm not always able to give a complete answer quickly enough. Some topics, such as this one, require experimenting, as I don't know the answer in advance. I'm physically unable to do a fair "round robin" over all the active ML topics during let's say one week, as that would require too many "context switches" for me (and I'm not good at that), so inevitably some topics are going to be delayed. Seeing a message (with new useful content) on some topic reminds me to look into it; seeing the same message in 3 different places... triggers a priority drop on my side. Others may react in different ways; no need to get mad at them just for that.

[Danne]

@a1ex +1

There is nothing personal about getting a little forum reminder now and then. I like the fact that Audionut is checking in on things. Keep up the good work everyone.

[reddeercity]

I did a head shot to see how bad with frame skipping enable is , usable but the frame drops are noticeable
had to cut up the audio a lot to make it sync up , it's still not 100% but close (be kind please)

I what to show the concept of near UHD (3k) 10bit raw video (thanks waza57  )

2880x1080 23.976p 10bit , EF 24-70mm @ 24x 1.95 crop factor = 48.6mm , 1/42th sec shutter , 400 ISO
mlv_dump to true 10bit  exported from Resolve 12.5 to rec709 DNxHR 444 , finished off in FCPX uploaded file ProRes422Hq
Hopefully this will encourage other 5d2/d4 cam user to join in

Since Youtube doesn't have 2k or 3k options it only play at 1080p
so I upload a mp4 to my goggle drive at the native resolution (2880x1080p) 600MB
https://drive.google.com/file/d/1obTz9AyEglhv9u3wjKS6x2mDWVxdkU79/view?usp=sharing

@a1ex thanks , but I feel I need to clear up something here first before I continue on (It's too important now to drop this development)
I think there's being a big miss understanding here ,
My original intent was not spam you with new threads & questions , My logic with to put all the CF card research for d4 cams in one place without
trying find little bit of info here & there and the UHS-I / SD cards investigation thread , to me that was the wrong thread to continue on
the discussion being it not cf card . So I could not understand why that first thread was locked , to me it was a road block for some reason towards me .
specially after I thought I made my self clear in the second thread I start with the same topic heading , and that turn in to a shit show as you know .

I still think a separate thread for CF card development is needed , but that my own opinion .
I'll just work from this thread for now on .
Sorry for the confusion .

[Audionut]

specially after I thought I made my self clear in the second thread I start with the same topic heading , and that turn in to a shit show as you know .

I believe that I need to set a standard, publicly.

http://www.catb.org/esr/faqs/smart-questions.html#keepcool

Much of what looks like rudeness in hacker circles is not intended to give offense. Rather, it's the product of the direct, cut-through-the-bullshit communications style that is natural to people who are more concerned about solving problems than making others feel warm and fuzzy................it may help you cope with our eccentricities if you think of us as being brain-damaged....

I attempted (I am not a robot, I am fallible) in the first thread to show some friendliness.

Sorry my reply wasn't useful.

Regards,
Audionut.

The second thread, was pure emotion.  It was made public, and it was dealt with publicly (see first line of reply in this post).  Taken to PM would have been significantly better.

I have critically maintained this public standard with you in the past, but having said that, at any point in time..........I could have also handled the matter in a more private nature, showing more respect to your contributions to the community.

Emotions take time to process from this keyboard, so I (unfortunately on some occasions) revert simply to,

Exaggeratedly “friendly” (in that fashion) or useful: Pick one.

I'm sure somehow, at some stage in the future, that I can be better at being useful, and also a little (more) friendly at the same time.  Maybe we could start some banter in general chat or something, where we can all just shoot some shit.  We could discuss topics in this thread, such as, hey Audionut, I'm trying to start a new thread regarding CF cards, not SD cards, you dumb shit!  Why you close my thread for?

[dfort]

I still think a separate thread for CF card development is needed , but that my own opinion .

A while back my opinion was that in order to get crop_rec_4k working on Digic 4 cameras we needed to figure out 10bit/12bit, lossless compression, get the basic crop_rec module working then finally tackle crop_rec on steroids and speeding up the SD/CD read/write speeds. Instead of trying to follow so many development discussions that had only a few posts related to Digic 4 cameras, I thought it might be better to keep all Digic 4 advanced development on this topic but I got chased off.

Hey, you did a great job with the theory and @waza57 figured out the code and posted a test build for 5D2 users. There are several other cameras that could benefit from your work and there is still a lot more to do before Digic 4 can be merged into the crop_rec_4k branch so let's not hold a grudge and keep the momentum going.

[reddeercity]

Find a little more info on CF cards , https://www.compactflash.org/cf-cards

Code: [Select]

83572>  CSMgrTask:ffbdb8a0:22:01: cfDecideTiming: UDMA Mode 6 (CFA4.0)So there the limit on the 5D2 , the card connects at udma6 @ CF4.0 = about 90MB/s

Does the 5D3 @ udma7 run at CF6.0 ?
Should be able to tell from a dm-spy log , I found mind after reviewing a cr2 file .
Can any 5d3 user confirm this please .

[Walter Schulz]

Does the 5D3 @ udma7 run at CF6.0 ?

I bet it does 6.0 or 6.1:

Screenshot by blitzkrieg666 in message https://www.magiclantern.fm/forum/index.php?topic=12630.msg161435#msg161435
Close numbers to firmware 1.1.3 as seen in https://www.magiclantern.fm/forum/index.php?topic=12630.msg144600#msg144600

[reddeercity]

Thanks  Walter , yes more then likely .
Found a very complete CF Card specifications pdf from 2005 , a little old (CF+ & CF SPECIFICATION REV. 3.0)
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/cfspc3_0.pdf
On page 67 there is Timing Modes definitions

Code: [Select]

83592>  CSMgrTask:ffbdbb3c:22:03: CF_GetAccessTiming : DatTim = 3, DatMod = 6From this it looks like "DatTim3" = 100 ns  , So does that mean 5d3 run in Mode 4 @ 80ns ?
I maybe wrong here , but it sure looks like it -- did waza57 test with udma7 on 5d2 need to adjust the mode time to shorter  e.g. 80ns ?
I my need to run some test in forced udma7 and see if I can get some logs

Edit:On Page 88-- noticed that CF cards can be run @ 8bit data & or 16bit data from the pdf , so maybe the d4 cams are in 8bit data write mode instead of 16bit data write mode
 

[Walter Schulz]

IMO (see page 2) all specs by CFA are copyrighted (outdated or not) and links to it might be able to harm ML site owner.

[reddeercity]

Ok , I put the PDF file on my bitbucket downloads & if that's not ok then I'll remove the link

[a1ex]

Some notes on fixing the live preview on 60D, after resolution overrides:

x5, x10:

C0F08518: 0x045109D7
C0F0851C: 0x00D601E6
C0F08520: 0x038A05FE

To find them: register diff with adtg_gui between two x5 configurations shifted horizontally by one position
(careful not to shift the captured image position; only the preview must be shifted).

Interpretation:
0x38A05FE - 0x00D601E6 => 1048x692.
"HD" YUV422 buffer size in these modes: 1024x680.

=> C0F0851C/C0F08520: output resolution, with some margins; these might be trimmed later.

Changing the lower half of C0F0851C and C0F08520 by the same amount: horizontal translation.
Same for the upper half: vertical translation.

C0F08518: 0x045109D7 = 2519x1105.
From C0F06084/88: 2520 x 1105. So, register C0F08518 specifies the "input" resolution (i.e. captured Bayer data). Subtract 1 on the X axis.

Increase X resolution by 1 unit from C0F06088: 0x45205B6 -> 0x45205B7 (1 unit = 2 pixels)

Fix the preview by adding 2 units to C0F08518: 0x45109D7 -> 0x45109D9.

Register C0F08518 is not used in 1x. It also appears in 640x480 crop mode, with value 0x026F0397. Raw resolution in this mode is 920 x 623. Check.

Register C0F08518 also appears on 5D2, 5x, with value 0x04670907. Raw resolution 2312 x 1127. Check. To fix the preview, C0F08188 has to be updated as well.

It also appears on 700D, CR2 capture, value 0x0DC80A4F. Raw resolution 5280 x 3529. Check, after adding 1 and multiplying by 2 on the X axis.

It does not appear on 700D in LiveView. It does not appear at all on 5D3.

Related: Bilal notes for 700D / mine for 60D: https://www.magiclantern.fm/forum/index.php?topic=9741.msg203674#msg203674

TLDR: looks like we've got a way to:
- specify the RAW (Bayer) input buffer size (for the image processing path that creates the live preview image in x5)
- pan the preview window around (without resizing it)

For image resizing: use adtg_gui to "diff" the registers between x5 and x10, then lock their values. Preview in x5 and x10 will stay the same. Figure out the meaning of these registers to find out how to apply arbitrary scaling to the preview image.

[waza57]

Please, What are the problems fixed here?

1 - The size of the preview (before or during the recording)?
2 - The preview is frozen or very slow ((before or during recording)?
3 - Bad previewing with artefacts (before or during recording)?

(On 5D2 the only issue (for me) is that the preview is too slow )

Thanks.

[a1ex]

That's right; CPU-based preview is slow, hardware-based real-time preview is... well... hard.

[reddeercity]

In the 5d2 & 5d3.113 rom disassembly found the same "cfDecideTiming" ,
Looking for the parameter that decide timing  & UDMA mode , still no luck yet
5D2 rom disassembly

Code: [Select]

ff3db894: e28f2f55 add r2, pc, #340 ; ff3db9f0: (65446663)  *"cfDecideTiming: UDMA Mode %d (CFA4.0)"
ff3db8b8: e28f2f56 add r2, pc, #344 ; ff3dba18: (65476663)  *"cfGetRegisterTiming: I/O %dnS (CFA3.0)"
ff3db8d8: 328f2e16 addcc r2, pc, #352 ; ff3dba40: (65476663)  *"cfGetRegisterTiming: I/O 250nS"
ff3db9a4: e28f20ec add r2, pc, #236 ; ff3dba98: (65476663)  *"cfGetRegisterTiming: I/O 120nS"
ff3db9bc: e28f20f4 add r2, pc, #244 ; ff3dbab8: (65476663)  *"cfGetRegisterTiming: I/O 250nS (PIO Mode4)"

5D3 113 rom disassembly

Code: [Select]

ff727f18: e28f2f56 add r2, pc, #344 ; ff728078: (65446663)  *"cfDecideTiming: UDMA Mode %d (CFA4.0)"
ff727f3c: e28f2f57 add r2, pc, #348 ; ff7280a0: (65476663)  *"cfGetRegisterTiming: I/O %dnS (CFA3.0)"
ff727f5c: 328f2f59 addcc r2, pc, #356 ; ff7280c8: (65476663)  *"cfGetRegisterTiming: I/O 250nS"
ff72802c: e28f20ec add r2, pc, #236 ; ff728120: (65476663)  *"cfGetRegisterTiming: I/O 120nS"
ff728044: e28f20f4 add r2, pc, #244 ; ff728140: (65476663)  *"cfGetRegisterTiming: I/O 250nS (PIO Mode4)"
Interesting 5d2 nearly Identical to 5d3 113 . There's no reference to any mode higher then "cfa4.0" on 5D3.

Edit: Looks little a closer (5D2 disassembly)

Code: [Select]

ff38affc: e28f2f7d add r2, pc, #500 ; ff38b1f8: (64496663)  *"cfIdentifyDrive: Set UDMA( Mode=%d )"
 (5D3 113 disassembly)

Code: [Select]

ff6aa9dc: e28f2f77 add r2, pc, #476 ; ff6aabc0: (64496663)  *"cfIdentifyDrive: Set UDMA( Mode=%d )"

[reddeercity]

I may have found the "CF card Configuration base"

Code: [Select]

RequestConfiguration: pLStorage=0x884e5c
81F67>  CSMgrTask:ffb8bcc8:22:01:                       ConfigBase=0x200, StatusReg=0x0
81F90>  CSMgrTask:ffb8bce0:22:01:                       PinRepReg=0x0, CopyReg=0x0
81FB6>  CSMgrTask:ffb8bcf8:22:01:                       ConfigIndex=0x1, Present=0xf
81FE2>  CSMgrTask:ffb8bd80:22:01: RequestConfiguration: Base = 200, Data = 41
8200E>  CSMgrTask:ffb8bde0:22:01: RequestConfiguration: SUCCESS
82031>  CSMgrTask:ffb8c688:22:01: CF_RequestConfiguration: err=0x0
82058>  CSMgrTask:ffb8c6a4:22:01: ConfigIndex = 1 Match
The "ConfigBase=0x200" sure looks very close to waza57 test

Code: [Select]

0xC062850C --> 0x1213 and I obtain UDMA 0 speed (around 15MB/s)
                     --> ...........
                     --> ...........
                     -->0x0202 and I obtain UDMA 6 speed (around 80MB/s) i suppose original valueSo could 0x200 be the same as 0x0202 or close , if so them maybe 0x100 will work instead 0x1000102 ? Just a thought 

[waza57]

I have not 5D2  ROM disassembly to compare but when i see this :

There's no reference to any mode higher then "cfa4.0" on 5D3.

I think it's a good news.

Could we think that "500" for 5D2 and "496" for 5D3 can be interpret as  parameters passed to function  "Set UDMA( Mode=%d )" ?

For 0xC062850C -> 100 , I have already tested and 5D2 freeze.

[aprofiti]

Found a very complete CF Card specifications pdf from 2005 , a little old (CF+ & CF SPECIFICATION REV. 3.0)

Also revision 4.0 (2006) can be found just by googling, but apparently no rev.4.1a neither 6.0

Instead latest revision can be bought from CFA website

[reddeercity]

@waza57 I sent you a pm , should help with post #269

[reddeercity]

Found some info on the CF5.0 specification , looks like it was implemented in 2010
Key Feature Benefits :

Code: [Select]

Capacity points beyond current limitation of 137GB (up to 144PB) (1,024 TB = 1 PeteByte)
& more efficient data transfer (32MB per transfer versus 128K per transfer)
Data Set Management Command/Trim (Mandatory)
More efficient cleanup of unused space on memory card (LBA’s)
Update ATA References to ATA-6 & ATA-8/ACS-2

Mr. Shigeto Kanda of Canon and the CFA chairman of the board said
“The higher capacity and higher performance of CF cards enabled by the 48-bit addressing feature in the CF5.0 specification
will further increase the value of DSLR cameras. The Video Performance Guarantee feature of the CF5.0 specification
will help CF cards to expand into new markets such as high-speed movie equipment like professional video camcorders”.

So it seems that there 48-bit addressing on CF5.0 vs CF4.0 32bit addressing
FAT File Allocation Tables https://en.wikipedia.org/wiki/File_Allocation_Table#FAT32
so can we trick 5d2 to cf5.0 for 48bit addressing , or is 48bit addressing only for exfat format ?
Still researching this .

Edit: Here the PDF file where I got the info about CF5.0
cf5_0_Spec100222.pdf

found the CF6.0 Spec PDF news release
cf6_0_101118.pdf
This seems important information

CompactFlash cards are capable of 600x or 90MB/second throughput performance using the current Ultra DMA Mode 6.
CF 6.0 Ultra DMA Mode 7 along with 48-bit addressing defined in the CF 5.0 specification enables the development of
CompactFlash cards with up to 25% more throughput performance.

@waza57 This looks very close to your 30% increase in write speed
So I guess we know why 5D2 is limited to around 79MB/s (UDMA6.0 CF4.0)

The CF 6.0 specification includes a new Sanitize Command, Trim Usage Guidelines and an Operating Temperature Range function
along with adding Ultra DMA Mode 7 which supports 167 MB/second speed.

Code: [Select]

Benefits: Ultra DMA Mode 7
Key Feature Enhancement:
Provides an interface speed of 167MB/s.
This speed enhancement enables a new generation of higher performance cards while providing complete backward compatibility.

Benefits: Sanitize Command
Key Feature Enhancement: Provides an efficient NAND Block Erase of
the entire user data area to return the CF card to “fresh” state before reuse or repurposing.
Leverages a command defined in INCITS T13 ACS-2.

Benefits: Trim Usage Guidelines
Key Feature Enhancement: Provides improved write performance consistency.

Benefits: Operating Temperature Range
Key Feature Enhancement: An optional card capability to report the
operating temperature range of the card.
This allows card/camera combinations to enable use in extreme temperatures.code]

So from this