Potential for All-I on 60D

Started by cthornhill, July 06, 2013, 12:59:55 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

cthornhill

Hope this is the right place...and if it is impossible, I understand...I see lots of exciting discussion of All-I on the 650D or 700D...and of course they have much faster SD cards than the 60D does (but the 60D does have good sized buffers at least)... I am using higher bit rates now on ML with my 60D and testing with raw, but higher bit rates in H264 have some advantages for me in some situations...much as I like raw... but I would love to get more than I am now, especially All-I if I could. I understand Tragic Lantern offers 650 and 700 users All-I, and wonder if that could be done on the 60D in ML...

In raw I can get a little better than 20MB/s, and that is around 160Mb/s - a lot more than I can see from X1.4 or even X2.0...it is really close to what I understand other testers of All-I on the 650D are seeing...so I wondered if such a feature might make it to the 60D...I don't expect to get audio with this (or course), but if we could get close to GOP=1 it would be pretty neat...

Of course I guess just buying a 650D, 700D or even the new 70D is another option, but getting more out of what I have would be pretty nice too...if it is possible.

Like I said - if it is not reasonable or worth it, I understand. Just hoping :-)

cthornhill

Time for a simplification...and a plea for data!

Last week I saw a thread between Jharding and 1% about the H264 controls in Tragic Lantern (1%'s experimental build). the reference is:

Quote from: jgharding on July 04, 2013, 06:19:57 PM
This post is designed to answer questions about the H264 codec control in one version of Tragic Lantern in a simple and practical way.

Jharding has been getting All-I H264 at high bitrate (up to about 160Mb/s) on his 600D. When I wrote to 1% about it he asked if any other people really wanted to do this on the 60D. I know I do, but I think the question is valid - is there much interest in this?

There is a bit of work to do to try and get the code working on the 60D (right now Tragic Lantern is a 600D project, but it does not run on the 60D yet). I agree that before anyone does too much work it  might be nice to see how much interest there is in such features on the 60D.

No one yet knows exactly how the 60D would perform. We have similar chips, more buffers, but a slower path to storage. On the other hand I know I get almost 20MB/s in raw, and that is about the same rate to disk need to replicate Jharding's results in Tragic Lantern (in theory).

If others want the advanced H264 options, please let it be known...I for one would love to be able to get 90Mb/s All-I if that is possible, and higher bit rates would just keep getting better. Maybe not as sexy as raw, but I think pretty darn useful and more economical than buying a new 70D for me...and I hope others.

Please help me answer 1%'s question - is there any interest in this in the broader 60D ML community?

PS - many thanks so far to Jharding, 1%, Marekk and others helping me investigate this a bit...I really appreciate your time and help/advice!

v8rrc

Yes Please Definitely interested.

sarotaz


gerk.raisen

+1 Definitely interested.
I can't wait to test it.

Thank you in advance 1%

albert-e

Quote from: cthornhill on July 06, 2013, 12:59:55 AM
Hope this is the right place...and if it is impossible, I understand...I see lots of exciting discussion of All-I on the 650D or 700D...and of course they have much faster SD cards than the 60D does (but the 60D does have good sized buffers at least)... I am using higher bit rates now on ML with my 60D and testing with raw, but higher bit rates in H264 have some advantages for me in some situations...much as I like raw... but I would love to get more than I am now, especially All-I if I could. I understand Tragic Lantern offers 650 and 700 users All-I, and wonder if that could be done on the 60D in ML...

In raw I can get a little better than 20MB/s, and that is around 160Mb/s - a lot more than I can see from X1.4 or even X2.0...it is really close to what I understand other testers of All-I on the 650D are seeing...so I wondered if such a feature might make it to the 60D...I don't expect to get audio with this (or course), but if we could get close to GOP=1 it would be pretty neat...

Of course I guess just buying a 650D, 700D or even the new 70D is another option, but getting more out of what I have would be pretty nice too...if it is possible.

Like I said - if it is not reasonable or worth it, I understand. Just hoping :-)

60D, per specification and the recent Raw video beta implementation exposes the truth about some Canon models and the CF manufacturers were out of specification regarding maximum transfer rate (in this case WRITE) and especially the Canon 60D, sadly the maximum write was 20mb/s. It shocks!(even the newer 6D).:-)) The only one with promise was the Canon 5D Mark III.


Can


Yuppa

Asked and answered awhile ago:

RE: "1%'s bit rate code."

From Marekk,

"I think this code is portable but I don't know how to find addresses for 60D."

So unless somebody is willing to find the addresses, as each body is different, it "ain't" gonna happen until it's incorporated in an official release (assuming it will be), not to mention (the devs) having to field a slew of "asked and answered" ?s regarding GOP and slice.

With the SD controller limitation of D!gic 4 bodies (re: RAW), this ALL-I H.264 (and GOP control) is where it's at and a no-brainer for 60D owners.  But as we (me...you?) are not devs, we have to beg and wait.

And yes, you can PM devs and demi-devs and ask THEM questions...dare I say, "pester?"
When you care more about capturing DATA, as opposed to WONDERMENT, you've lost your creative SOUL.


cthornhill

Hi - I am supposed to know about how to find stuff in ROMs, but it has been years since I did it...also I am pretty swamped at work. I do have the 60D code in my disassembler (I am using PEL's free tool) and got some output from ROM0 and ROM1 on the 60D. I have not had time to look at the stings and assembler much and it will be a few days given work issues.

I am not yet sure what I am looking for (exact strings /op codes/ addresses)...so I need to get some background and probably look at 1%'s code too (that will take more time).

1% suggests I try the arm console, and I need to get that set up on a workstation too (not done with that yet, nor do I have the app toolchain installed to compile yet)....

I want to help, but have to do only what I can in the time I can give it...sorry.

If anyone has suggestions, instructions, or wants data from me, let me know. I will do what I can, but I have to balance this with some pretty heavy commitments at work. Alas I have been away from day to day coding for long enough that I am very slow at this so if anyone wants to go faster and needs resources I have I will do what I can to help. Otherwise, maybe we will get there in time.

Glad to hear others have interest.

BTW - Check out the results Jharding is getting on the 600D in his threads on the 'shoot preperation' forum section...interesting stuff and what inspired me to want to try this on the 60D.

Can

Quote from: cthornhill on July 11, 2013, 05:01:55 PM
I want to help, but have to do only what I can in the time I can give it...sorry.

That's nothing to apologize for - thanks for even considering doing this. We get a little greedy with all these exciting new developments and are always looking for more and better.

If we can get the addresses we are looking for and it would help if I were to take on some leg work and comb through the ROMs, just let me know - I'll be glad to help.

cthornhill

1%  and marekk have had some conversations...1% suggested we need to find the addresses for his Tl v1. code. Marekk found the sections in question (we think) for the 600D at:

https://bitbucket.org/OtherOnePercent/tragic-lantern/src/d2f9724dc97589eefdb93aad6579f50ccd69ac24/src/bitrate-600D.c?at=unified

I believe we should look at the source, identify the codes/addresses and then go through the 60D code and see if we can find those corresponding addresses in our 60D code. I am assuming we can search for the stings and fine the hex address that matches since it appears the code just pushes settings to registers by address so the camera can operate on them...at least that is my rough assumption.

I am just starting to look at the code from 1%, but any help is welcome. I will then try to see what I can find in the ROM that might look like it matches up...again, just getting into this myself.

cthornhill

Not sure if this is all we need...but for any help it may be, I exported a text file of the addresses/strings I got from the dissasembler (Pel's).

This is not quite the same as the view inside the tool, but I think all the strings are here. I only had time for a few min. looking as I have to get back to work, but I found lots of promising addresses for GOP values (if I am reading the strings right). Lots more to check, but if you want to see the file it is at this location:

https://www.dropbox.com/s/kyo7nw9xq7l85q6/test.idc

This is NOT the Canon ROM (that is their property) but is data about some stings and addresses I found in that ROM...so we are in a gray area...but if it is OK to use in source (the hex) then I think I am OK, but I will trust the admin's to let me know if posting this is OK. If not, they or I can take it down.

Cecil

Bioskop.Inc

This would be v.useful, as raw is only really useful if you use an anamorphic lens & stretch the image. 

However, i did find something that was for the 60D on 1%s download list, but it seems to have gone!
It was an autoexec.bin that had a v.basic extra bitrate hack with GOP etc...

https://dl.dropboxusercontent.com/u/90827273/autoexec.bin

It loads fine, but I haven't tested it out properly, but could it be a first step in the right direction?

cthornhill

not sure...I did not see a TL version for the 60D but may have missed something. I think the current code on that project is 600D...but 1% did say it should be portable. I will try and look again, but I think we will need a new build with data on addresses on the 60D ROM's for the code to run right.

Bioskop.Inc

Quote from: cthornhill on July 11, 2013, 08:02:13 PM
not sure...I did not see a TL version for the 60D but may have missed something. I think the current code on that project is 600D...but 1% did say it should be portable. I will try and look again, but I think we will need a new build with data on addresses on the 60D ROM's for the code to run right.

Try the autoexec.bin i've supplied, at least its something for the meantime (it was in 1%s download section a few weeks ago, but isn't there anymore - i checked as well).
Also, someone will probably be able to open it & see what's what - it might provide the addresses.

cthornhill

Thanks - I will when I get a chance - may be tomorrow...long work session starting shortly (midnight oil and all that)...I will let you know what I find out. I assume I need Tragic Lantern and this autoexec to see what happens - any specific build of Tragic...or am I wrong and this somehow is useable with a version of ML (if so which one?)

REAL STUPID...sorry. Of course - install ML then replace the autoexec...sorry, my brain stopped working for a moment there... ;)

Like I said - I will give it a try...and let you know.

At least it does appear that lots of people are starting to get some enthusiasm for high bit rate/All-I which is what I suspected would be the case...I do think there is some demand...good as raw is (and your raw tests are impressive in particular) I think there is going to be lots of need to keep improving H264 too on many cameras for some situations.

Got to run - will post test answers when I can.

cthornhill

Bioskop.Inc - good news and bad news. I stopped for a bit to just run a fast test to see what happens when I boot - not a lot of time to shoot just now.

Good news - the Autoexec boots and is clearly a version of TL for 60D.
Bad news - this version does not have the slice CBR settings needed.

There is an advanced Encoder options menu but not the options now found in the 600D version as used by Jharding so I could not follow his post on 'Shoot Preparations' to try a test.

It has a lot of the older settings (many of which may equate to the newer ones)...but I really think I need the current version of the code for a meaningful test, so I am not planning to mess with the older settings just now. I think it may be just as well to keep pressing on with the ROM address data as 1% suggested. I have lots of that now, so I think I am going to press for that.

Many thanks for the autoexec. I will keep it and if I feel adventurous I may look at it more to see if I can decipher a relationship in the old menu settings to the new ones. For what it is worth the old menu options in this version are:

ENCODER OPTIONS
P factor
I factor
D1 factor
D2 factor
Gop0 factor
Gop1 factor
Gop2 factor
Gop3 factor
Gop4 factor

All default to 1.0x

I will dig out my notes on H264 and look at the older posts to see how these might match up to the new settings which are ( so I assume per Jharding's post - I don't have a D600 now):

IN SLICE CBR

Lock Slice: Disabled
Min BR: 130
Max BR: 160
Drop by 1: 140
Drop by 3: 145
Taper Rates: Enabled.

IN BITRATE MENU
Mode: CBR
DblockA: -6
Dblockb: -6
PicPC: 0
GOP: 1
Bitrate Info: On (I find it interesting)
BuffWarnLevel: 70% (just in case)

From my memory of the H264 code process (long time since I hacked compression) slice is pretty intimate with macroblocking. My guess is that DblockA and Dblockb in the new code map to D1 and D2, and I am not sure what is up with the 4 Gop's, but I know we want to get to GOP of 1 for All-I...Like I said, when I have more time I can see if there is a map that works...but I really want to just get to the new code so we can test apples to apples on the 60D and 600D....but hey, this is a start.

I don't want to appear ungreatful - this is a helpful thing, but maybe not the final step. I really do appreciate it. If I get some video I will pass on the results...but that may be a while.

Thanks Again!

cthornhill

To all - With some suggestion from marekk I found the source file for 600D new bitrate controls:
https://bitbucket.org/OtherOnePercent/tragic-lantern/src/d2f9724dc97589eefdb93aad6579f50ccd69ac24/src/bitrate-600D.c?at=unified

In here it lists many values that I believe we will need to find in the 60D rom, such as:
#define UNK_GOP_LOC 0xFF25516C
#define GOP_MISMATCH_LOC 0xFF048054
#elif defined CONFIG_1100D
#define UNK_GOP_LOC 0xFF2478E8
#define GOP_MISMATCH_LOC 0xFF047E4C

I am still getting a feel for this, and I see lots of similar data in the 60D ROM's I have but much does not match 1:1...for instance I have found at least addresses such as:

MakeNameEx   (0X1AE274,   "amvrFixQScaleFI", SN_CHECK);
   MakeStr   (0X1AE2A0,   0X1AE2D6);
MakeNameEx   (0X1AE2A0,   "amvrSetPrintMov", SN_CHECK);
   MakeStr   (0X1AE2D8,   0X1AE2E4);
MakeNameEx   (0X1AE2D8,   "amvrSetQscale", SN_CHECK);
   MakeStr   (0X1AE2EC,   0X1AE2FA);
MakeNameEx   (0X1AE2EC,   "amvrSetQscaleYC", SN_CHECK);
   MakeStr   (0X1AE300,   0X1AE316);
MakeNameEx   (0X1AE300,   "amvrSetDeblocki", SN_CHECK);
   MakeStr   (0X1AE31C,   0X1AE32D);
MakeNameEx   (0X1AE31C,   "amvrSetLimitQSc", SN_CHECK);
   MakeStr   (0X1AE330,   0X1AE33F);
MakeNameEx   (0X1AE330,   "amvrSetDefQScal", SN_CHECK);
        MakeStr   (0X1AE340,   0X1AE34F);
MakeNameEx   (0X1AE38C,   "amvrSetGopOptSi", SN_CHECK);
   MakeStr   (0X1AE3A4,   0X1AE3B6);
MakeNameEx   (0X1AE3A4,   "amvrSetGopOptSi", SN_CHECK);
   MakeStr   (0X1AE3B8,   0X1AE3CB);
MakeNameEx   (0X1AE3B8,   "amvrSetGopOptSi", SN_CHECK);
   MakeStr   (0X1AE3CC,   0X1AE3DA);
MakeNameEx   (0X1AE8E0,   "aOptGop0d1d2d3d", SN_CHECK);
   MakeStr   (0X1AE914,   0X1AE940);
MakeNameEx   (0X1AE914,   "aIniQScaledIniD", SN_CHECK);
   MakeStr   (0X1AED10,   0X1AED25);
MakeNameEx   (0X1AED58,   "aGopSized", SN_CHECK);
   MakeStr   (0X1AED68,   0X1AED88);
MakeNameEx   (0X1AED68,   "aIOptSize2dPOpt", SN_CHECK);
   MakeStr   (0X1AED8C,   0X1AED9B);
MakeNameEx   (0X1AED8C,   "aQScaleddd", SN_CHECK);
   MakeStr   (0X1AED9C,   0X1AEDCF);
MakeNameEx   (0X1AED9C,   "aQScaledDBFAdDB", SN_CHECK);
   MakeStr   (0X1AF030,   0X1AF041);
MakeNameEx   (0X2B8108,   "aGOPd", SN_CHECK);
   MakeStr   (0X2B8120,   0X2B8137);
MakeNameEx   (0X2B9E4C,   "aGOPNextSet", SN_CHECK);
   MakeStr   (0X2B9E5C,   0X2B9E77);

and so forth....in short, I have to get some time to match the C and the ROM assembler before I am comfortable knowing that I am looking at the address that represents the one I really need...as I am sort of diving into the deep end of the pool here.

If anyone is more used to 60D ROM and extraction, and they want any of this to go faster let me know.

I feel like I am starting to see all the parts, but need to get a lot deeper into the way they fit together. Also I am trying to read through the pages of the work so far on this option in TL...lots to absorb...small old brain to do it with...:-)

Also - if we need to just take up a collection to get a 60D body to a more experienced developer, I at least am OK with that...I will keep plugging slowly, but I am not ashamed to let others who are better or faster take it up, and I will help with that as I can, including donations...:-)

Can

Cecil -

This is a great start - your enthusiasm is inspiring and very much appreciated. Thanks!

Sadly, I couldn't hack my way out of a paper bag...but I can certainly test. ;)

marekk

Code is the same. I used disassemble.pl from chdk and I've found all adresses. I will try to compile it tomorrow and test how it works.

cthornhill

Marekk - you are my hero! Thanks...I was up most of the night with a restore (from a failover earlier this week) so I am fried. I did spend some time while waiting on the line last night to read most of the old thread that stated this...I can see how they evolved from pushing the CODEC to RAW...and I am starting to see how more of the registers are being used...it has been a while since I messed with CODECs and compression in general (back in my medical imaging days).

In general I am interested to see what we can get in terms of a data rate, and what that translates to in terms of images. Like Jharding I have a Mosaic filter, and it helps with IQ by reducing chroma noise from aliasing but so far I see some improvement even at X1.4 (about 64 Mbs) in IQ and I really hope that if we can get close to ALL-I rates (say 90-120 Mbs) it will be even better. There is still chroma trouble due to 4:2:0 (like edges of saturated objects in frame, and general chroma aliasing on finer detail), but not a lot we can do for that just now in H264.

Jharding uses deep noise reduction, and I want to try that too, and maybe in intermediate format like 4:4:4 transcode...but all that is in the future.

Anyway I would love a build, and when I can I will keep plugging on my toolchain to try to get to the point I can compile it too. For now I really appreciate your doing the heavy lifting!

THANKS AGAIN!

cthornhill

Hey Marekk - can you help me with the steps you used to get the CHDK tool working...not sure what address to use with their debugger for instance...I want to set up those tools too if I can...if that is too much no worries. I will check the disassembly threads too.

marekk

I replaced original bitrate.c code with 1% Tragic Lantern code with some modifications for 60D. There are some problems with compiling because something changed in menu.c and menu.h and now there is no "display" function.

../../src/bitrate.c:1778:17: error: unknown field 'display' specified in initializer
../../src/bitrate.c:1778:17: warning: initialization from incompatible pointer type [enabled by default]
../../src/bitrate.c:1778:17: warning: (near initialization for '(anonymous)[1].select_Q') [enabled by default]


When I removed .display from menu entries it compile and firmware starts on the camera but all values aren't displayed properly.

bitrate.c
https://docs.google.com/file/d/0B-HdscXfsKpgVy1WUnRSVGxXS1k/edit?usp=sharing

cthornhill

so I guess we need a little help from 1% on cleaning that up....and I need to get my compile chain working to catch up...:-)

thanks - hopefully this will get into sync pretty soon as we can get useful test results.

I will work on my compiler issues per your other note to me. - Thanks.