Author Topic: Danne's crop_rec_4k & ISO experiments  (Read 10041 times)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #25 on: November 14, 2018, 05:05:17 PM »
Might very well be. SOrry but what are the timer A+B original values?

Tried this but not really helping:
Code: [Select]
#elif defined(CONFIG_100D)
    #define TG_FREQ_BASE 32000000
    #define FPS_TIMER_A_MIN (fps_timer_a_orig)
    #undef FPS_TIMER_B_MIN
    // no need to cause confusions as recording speed cannot handle such high fps in crop mode
    // (ZOOM || MV1080CROP ? 1288 : 1970)) <-- these are ok while not recording.
    // Hybrid CMOS AF II uses 60fps by default in LV/MV for the camera display
    // to achieve a "snappy" autofocus by doubling the fps
    // MV720 is not LV so we need to extend the definition for the LCD.
    #define FPS_TIMER_B_MIN (fps_timer_b_orig)

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 1139
Re: Danne's crop_rec_4k & ISO experiments
« Reply #26 on: November 14, 2018, 05:14:08 PM »
fps_timer_a_orig
fps_timer_b_orig
70D.112 & 100D.101

IDA_ML

  • Hero Member
  • *****
  • Posts: 642
Re: Danne's crop_rec_4k & ISO experiments
« Reply #27 on: November 14, 2018, 05:28:48 PM »
Hello all,

I was finally able to borrow a 5D3 for testing the latest very exciting 1x3 recording developments but only for a very short time - I have to return it tomorrow. Nonetheless, I thought, it might help if I provide some feedback on what I was able to test in that short time.
 
I tested briefly the two 17fps options in Danne's build:

https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3113_presets-ver2.zip

and all I can say is that this is another break through for Magic Lantern.  I would have never dreamed of shooting at 400/3200 Dual ISO at 1,0 crop factor and get usable results.  Not only are they usable and quite clean, but  also look very natural compared to previous techniques for improving dynamic range which in most cases provided very HDR-ish looking images.  In my opinion, this 1x3 technique, in combination with Dual ISO, provides a radical solution to low-light shooting high-quality FHD video in RAW without or with very little noise in the shadows.  I don't think, too many of the professional cinema cameras can do that.  Congratulations to all of you making this possible!

Also, I greatly appreciate Danne's efforts to provide continuous recording options, even at the expense of reduced frame rate.  There is nothing more annoying than early camera stops in the middle of a shooting.  And I don't think that 17 fps is that much of a problem.  Human eye accepts everything above 12 fps as motion and earlier silent movies were shot at 16 fps.  I shot a wedding film recently that I  filmed at 18 fps and nobody noticed that.  Nowadays we also have quite good optical flow techniques.  Of course, if you do a lot of camera motion, (slider, gimbal, flycam work, etc., requiring fast camera motion) you may end up with some choppy results but there is a solution to that too.  Motion should be slower for 17 fps, compared to 24 fps, and as smooth as possible, so training is required.  If you shoot from a tripod, monopod or even handheld and do not shake the camera too much, results are indistinguishable from 24 fps and the motion blur of moving objects in the scene really looks nice.  So, 17 fps is not necessarily a bad thing!

I didn't have time to test all the options, especially the 10/12-bit ones at different ISOs in the crop menu - this requires a lot of time and very systematic efforts to pixel peep all combinations and select the ones that provide the cleanest results with the highest dynamic range.  I hope that other 5D3 owners will jump in with more test results very soon which will be of great help to the developers.  However, I did test some of the high-resolution crop modes (I use them all the time, especially with the 100D) and am sorry to say that they are all broken in the above build.  The 3k crop mode that I have been successfully using to film the wedding at 3072x1728 resolution and 18 fps does not seem to be working at all.  It does not allow me to set any resolution but stays all the time at 0x0.  The 4k and Full-resolution Live View modes seem to be kind of working but the results are horrible - a lot of aliasing along horizontal and vertical edges, noise and loss of resolution - both, in the normal and Dual ISO modes.  Camera operation in those modes is very unstable either - lots of camera freezes requiring battery pulls. These high-res crop modes used to work very well even in the very early builds from April 2017. 

At this time I would make the following suggestion to A1ex and the other developers:  Please try to implement working 17 fps presets from Danne's build above into one of the experimental 5D3 builds.  Please use the July 4-th build #78 that you can find here:

https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/

The later ones, including the July 22-nd build, do not seem to be working properly - lots of instabilities.  Once such an experimental build works properly and is stable enough, I am sure many professional 5D3 video shooters will start working with it and will greatly appreciate it.  This is because they will have a very powerful tool in their hands that they will use for a long time in their work.  This is the dream of every video shooter, including me.  I would like to see a build that can provide the following modes:

1) Normal 1,0 crop factor FHD (1920x1080) RAW video with maximum quality and minimum noise that can be used in low-light conditions with Dual ISO.  The 17 fps presets already do that;

2) 3x-crop FHD (1920x1080) RAW video (1:1 mode) that will allow me to quickly change the focal length of my lens from 1 to 3x on a push of a button without changing lenses or using heavy zooms.  A 35/F2 IS lens is perfect for that and I use it all the time switching from 35 to 105mm focal length.

3) A 3k mode that I can use at 3072x1728 resolution and 18 fps for continuous recording (also with Dual ISO).  This one is perfect for capturing scenes with a lot of fine detail (group shots with 50 or more people on them, landscape videography, etc.).  Here is a film of the ancient city of Plovdiv that I made recently and that was filmed in that mode with the #78 build:

https://cloud.mail.ru/public/Grpx/iLK4KHaKi

(To watch it in the highest quality, please download it with the "Скачать" button from the top left of the above link).

4) A Full-resolution Live View mode with up to 5 fps continuous recording for smooth timelapses.

5) The usual 50/60 fps modes

For the beginning, 14-bit lossless compression would be perfect but wherever 13, 12 or 10 bit lossless could be implemented, would be very welcome to reduce file size.

And one final thought. Now that we have all the above functions more or less working and ready to use and we are close to the ultimate quality ML RAW video, in my opinion, future research should be directed towards increasing the interface bandwidth for higher frame rates and/or higher continuous recording resolutions.  This can be done in two ways:

A. Reimplementing card spanning on the 5D3.  I don't know why this extremely useful function was removed;

B. CF-card overclocking in a way similar to SD-card overclocking on the 100D - a God sent function.

It also would be very useful to create similar builds for other camera models such as the 100D, 7D, 650D and others.  The 5D3 is still quite expensive and the majority of ML users, including me, use the other camera models.  We all would love to see a long time awaited update to our cameras too in a way similar to what Bilal did with the 700D already. 

Please let me know what you think about my suggestions.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #28 on: November 15, 2018, 03:14:31 PM »
Thanks for feedback IDA_ML. I checked and surely a lot of presets were broken. I readded all my presets to the crop_rec.c from crop_rec_4k branch and added it to my branch instead. Seems to work better:
https://bitbucket.org/Dannephoto/magic-lantern/commits/8846065dfd242f82de3942cc3295f5f33225f7e4

Downloads:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3113_presets.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3123_presets.zip

In addition to 1x3 presets and isostuff in the submenu following is added:
- Added changes for 1920x1080 48 fps
- Changed following registers:
Code: [Select]
adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (5D3/6D/700D) adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (5D3) */

One very annoying bug is when camera mode is set to mv720p. Zooming will require battery bull when trying to get out of x10 zoom mode after previewing. No idea why it gets stuck. Happens with crop_rec presets.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #29 on: November 16, 2018, 01:52:38 PM »
In addition to the previous bug I also noticed that "framing" preview isn't working when hdr is set. Anybody can confirm? Known issue a1ex? Really useful working when using 1920x1080p 48fps.
Also uploaded builds which should be more stable shooting with above mentioned setting.

IDA_ML

  • Hero Member
  • *****
  • Posts: 642
Re: Danne's crop_rec_4k & ISO experiments
« Reply #30 on: November 16, 2018, 04:19:21 PM »
Danne,

I now remember one more bug that I noticed when I had the 5D3 in my hands but just forgot to report it in my post #1954 above.  It concerns "Full-resolution LiveView: 5796x3870 at 7.4 fps" which is very useful for shooting  smooth timelapse sequences at low frame rates using the full sensor resolution.  When activated, this mode records just one single frame and then camera stops.   This behavior is observed at any resolution set.  In older builds (those from April, 2017) this Full-resolution LiveView mode was working just fine, providing continuous recording at 5 fps as also stated on the Experimental page.  A fix should be greatly appreciated.

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1476
  • 6d - Nightly build user
Re: Danne's crop_rec_4k & ISO experiments
« Reply #31 on: November 16, 2018, 06:03:58 PM »
Now you mention it, I've made a full resolution live view crop preset for 6d.
And it behaves the same as the 5d3, only one frame is recorded.
The 6d has about 250Mb of free buffer, so you would at least expect to fit about 7 uncompressed frames in that buffer ??? (Not sure when lossless kicks in, is that before or after the buffer ?)


norek

  • New to the forum
  • *
  • Posts: 8
Re: Danne's crop_rec_4k & ISO experiments
« Reply #32 on: November 16, 2018, 08:26:24 PM »
Hi Danne,

Can I use, your latest crop_rec.c to eos m if I build your "crop_rec_4k_mlv_snd_isogain_1x3_presets" branch build to eos m?

thank you for Your hard work! :)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #33 on: November 16, 2018, 09:02:05 PM »
For eosm it's the same presets as with crop_rec_4k.

IDA_ML

  • Hero Member
  • *****
  • Posts: 642
Re: Danne's crop_rec_4k & ISO experiments
« Reply #34 on: November 17, 2018, 02:52:41 PM »
Danne,

I have been inspecting the files that I shot on the 5D3 with the latest 1x3 crop modes recently and found that now the 5D3 is one BIG step closer to perfection.  The Crop ISO (CISO) settings work so well in providing clean shadows that the results are really amazing.  Here are a few more minor issues that need to be fixed to get the dream 5D3 video shooting build:

1) The 6400 CISO setting still has vertical stripes in the shadows.  If these could be fixed and just the noise is left, this high CISO setting may be a life savior in some really dark situations, especially keeping in mind that Resolve does a pretty good job with noise cleaning.  Please note that the CISOs 3200 and lower settings are perfect already - no vertical stripes at all.

2)  MLVApp does not recognize and report the correct CISO at which the clip was shot.  It reports Dual ISO 100/800 all the tme, regardless of what CISO was used.

I also have a few questions:

I.  We now have the possibility to use the normal FHD MLV RAW video mode with Canon's ISOs and the mv1080p_ mv720p mode with CISO and Canon's ISOs.  What exactly is the difference between these two modes and does one of them provide advantages vs. the other in terms of video quality (fine detail, dynamic range, etc.)?

II.  Do the CISO settings apply to all crop modes in the Crop menu?  Will they clean the Shadows of the high-resolution modes in the same way and strength as they do with the mv1080p_ mv720p mode?

III.  I have noticed that it is possible to activate several CISO settings at the same time - e.g. the 200 and the 1600 setting.  Which one will be applied in this case?  Could these clean CISOs be used to also make Dual ISO with two CISO settings simultaneously active?

IV.  This question is for A1ex and is actually a tool request.  Would it be possible to add an indicator showing how much space is left on the CF card while recording?  In the 1x3 crop modes the 5D3 eats card space for breakfast and you never know when you will run out of card space.

V.  I understand that there are problems with overclocking the CF-card interface.  Would it be possible to overclock the SD-card interface instead?  If it could be overclocked to say 30 MB/s and card spanning could be applied, then the 5D3 would write at a speed of about 150 MB/s and this would significantly increase continuous recording times at the high-resolutions and continuous recording at 24 fps at the 1920x3240 1x3 mode would be possible.

VI.  Danne, could you please provide some more information on how to use the "Shutter range" and the "Advanced options" in the Crop menu?  In what situations should those be used?

Thanks.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #35 on: November 17, 2018, 05:42:07 PM »
Since my iso experiments was built on faulty assumptions about how analog gain registers and analog iso registers were related I decided to simply erase those code parts in following builds:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3113_presets.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3123_presets.zip

What still is valid testing and developing is 1x3 presets together with playing around with analog gain registers creating 10/12 bit. Those two settings can still be found in the sub menu of crop mode.
In a longer perspective iso tweaking is still valid and could be developed further I´m sure. I might dig into this and revive some code parts in crop_rec later on but without mislabeling menu items.

Commit:
https://bitbucket.org/Dannephoto/magic-lantern/commits/4e18649356067e49e15134e432025acdb2b6f0a7

jimiz

  • Freshman
  • **
  • Posts: 59
Re: Danne's crop_rec_4k & ISO experiments
« Reply #36 on: February 15, 2019, 03:01:55 PM »
Since my iso experiments was built on faulty assumptions about how analog gain registers and analog iso registers were related I decided to simply erase those code parts in following builds:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3113_presets.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3123_presets.zip

What still is valid testing and developing is 1x3 presets together with playing around with analog gain registers creating 10/12 bit. Those two settings can still be found in the sub menu of crop mode.
In a longer perspective iso tweaking is still valid and could be developed further I´m sure. I might dig into this and revive some code parts in crop_rec later on but without mislabeling menu items.

Commit:
https://bitbucket.org/Dannephoto/magic-lantern/commits/4e18649356067e49e15134e432025acdb2b6f0a7


@Danne

Can you compile crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Feb14  also for 5d3 123 ?

thanks lot !!!
5D3-123

jimiz

  • Freshman
  • **
  • Posts: 59
Re: Danne's crop_rec_4k & ISO experiments
« Reply #37 on: February 15, 2019, 05:02:54 PM »
THANKS!!!   :)
5D3-123

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #38 on: May 23, 2019, 03:09:47 PM »
Continueing from here:
https://www.magiclantern.fm/forum/index.php?topic=19300.msg216898#msg216898


Worked on my 5D3 crop_rec.c code in this branch:
https://bitbucket.org/Dannephoto/magic-lantern/branch/crop_rec_4k_mlv_snd_isogain_1x3_presets

Commits:
https://bitbucket.org/Dannephoto/magic-lantern/commits/c89688d60f2c0da642c9460deb1cf1cf79553148
https://bitbucket.org/Dannephoto/magic-lantern/commits/0d200f7cb6d6158d83d38c9092e4bcdef208d76f

- Refined anamorphic presets so it now gives continuous:
1808x2300(2:35.1) = 5425 x 2300
1360x2300(16:9)    = 4080 x 2300

- Also adressed the battery pull issue happening when in 48fps or mv720p 3x3 mode. Workaround allows for looping through x5,x10 and back into x1 mode.

- cine.lua included
- set_25fps in crop mode sub menu working with anamorphic mode
- bitdepth modes in crop mode sub menu working for all presets
- not included sd_uhs(don´t see why sd patching would be needed with mlv_lite anyway)

Downloads:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Oct15.5D3113.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Oct15.5D3123.zip

In terms of image quality and performance the anamorphic mode on this camera outruns anything coming from eosm, 100d etc...

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #39 on: May 23, 2019, 03:23:09 PM »
@Danne
modules loaded: crop_rec, mlv_snd, mlv_lite, mlv_play
res: 1920x1080p /14bit lossless (not crop mode)
CF (KB1066x) card seems to be working/saving fine when I revert to a previous 4K_crop-rec Danne build 2/4/19 :)

Still odd…
Try fomatting card in camera (non exfat) reinstall clean build.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #40 on: May 23, 2019, 03:24:37 PM »
now it works im not sure what i had done in my first test. now i just clean install and turn on the anarmophic preset in crop rec and no more greenish black preview. i have a very slow 4gb cf card atm. wil test more as soon a i recieved another card
Yes, bleeding edge stuff. Faster card will probably help. I set regs all over the place. A safety press on menu button and thrashcan once in a while to settle all regs won´t hurt too.

timbytheriver

  • Senior
  • ****
  • Posts: 302
Re: Danne's crop_rec_4k & ISO experiments
« Reply #41 on: May 23, 2019, 03:40:39 PM »
Try fomatting card in camera (non exfat) reinstall clean build.

Tried this. Get a Canon menu error:

Cannot format
Change card1

Yes, am saving to the CF card (1)

Still getting CARD FULL in top left of ML menu when I press record in any res. My CF card has been faultless til today.

Files into MLV App produce error: 'File header is missing: invalid MLV" or "file length too short Invalid MLV'

5D3 1.1.3
5D2 2.1.2

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #42 on: May 23, 2019, 03:43:32 PM »
Sorry to hear but I suspect this is the reason but I can´t say for sure why it would break your card:
https://www.magiclantern.fm/forum/index.php?topic=19300.msg216884#msg216884

Tried formatting your card to fat32 on computer and then format in camera?

timbytheriver

  • Senior
  • ****
  • Posts: 302
Re: Danne's crop_rec_4k & ISO experiments
« Reply #43 on: May 23, 2019, 03:44:48 PM »
That I copied only the crop_rec.mo file?

I've done that before without issue…

Have I tanked something?
5D3 1.1.3
5D2 2.1.2

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #44 on: May 23, 2019, 03:45:29 PM »
Tried formatting your card to fat32 (or fat, can´t remember exactly what it´s called) on computer and then format in camera?
Well, I can´t say what would work or not here. It´s all guesswork atm.

timbytheriver

  • Senior
  • ****
  • Posts: 302
Re: Danne's crop_rec_4k & ISO experiments
« Reply #45 on: May 23, 2019, 03:47:00 PM »
Tried formatting your card to fat32 on computer and then format in camera?
Well, I can´t say what would work or not here. It´s all guesswork atm.

Yes! See four posts above. :)
5D3 1.1.3
5D2 2.1.2

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #46 on: May 23, 2019, 03:48:18 PM »
Your answer indicates formatiing in camera but not on computer to fat. Do you have another card you can test my build on? If that´s working clearly your card is having an issue.

timbytheriver

  • Senior
  • ****
  • Posts: 302
Re: Danne's crop_rec_4k & ISO experiments
« Reply #47 on: May 23, 2019, 03:49:50 PM »
Oh right, sorry. Did you mean FAT32 instead of exFAT? Thought it needed to be exFAT?
5D3 1.1.3
5D2 2.1.2

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5906
Re: Danne's crop_rec_4k & ISO experiments
« Reply #48 on: May 23, 2019, 03:52:20 PM »
Oh man, we need @Water Schultz on this one. Find one of his rescue procedures and see if that will work.

timbytheriver

  • Senior
  • ****
  • Posts: 302
Re: Danne's crop_rec_4k & ISO experiments
« Reply #49 on: May 23, 2019, 04:11:30 PM »
I'm going to revert to previous build and see if I can get back the status-quo!
5D3 1.1.3
5D2 2.1.2