3K/UHD 5D2 Raw development and Other Digic IV Cams

[Igor_Braun]

Today when i was shooting a video without the crop mode for youtube it came again. Vertical lines and something strange with colors... it disappears with time.


 
Iso 400
WB 5400
https://drive.google.com/open?id=1GaTK9QzHNCzCmXwiMNXKOxxuQrFHw8ms
If it necessary i can upload the MLV

[Galterius]

I used the last crop_rec module and exported in h.265 (2560x1200), and colors change while I am recording, image is green at the beginning, and then it becomes normal (plus there is a lot of color noise). (it is out of focus at the beginning because of me ^^ but we can see some weird lines); On some clips I don't have this problem at all...
I will try with an older crop_rec.

And most of my clips have a very green tint
here it is : https://drive.google.com/open?id=1DDOEYHpd3js1NWW8jdeKUp0cFdbLr7-5

I can upload the mlv if needed.

[Danne]

Probably power timing regs needs tuning. Code, code, code...

[Galterius]

i just tried the crop_rec from the old version (it does 3008x1080) and it seems to work

[reddeercity]

Ok thanks for the results , it looks like the real time preview reg's are the issue .
I'll look in to this further , later tonight I'll post a non real time preview for 2600x1200
so it can be used until I fix the problem .

[reddeercity]

Ok let give this new test crop_rec 2.6k_5d2_R_T_PreView_4-7-2019_crop_rec.rar a try .
I re-worked the preset a little more , reduced the raw horizontal resolution from 2632 to 2600 for a final of 2592 in mlv_rec (mod8) so same output as before
just different way of getting that same size & re-adjusted the raw preview Reg's . The horizontal raw preview reg was missed matched to the total raw width with the OB area .   


Worked best with grey scale preview ( still have a issue with full canon color) it's still realtime (buy the way) but in B/W .
I tested it at 800 ISO & 1/64th shutter speed at 29.97fps in 3x3 (FHD) when I entered in to crop_rec for 5x zoom
the shutter speed changed to 1/48th of second   ( that's by design  )

Also check it with the HDMI EVF , @ 480p & 1080i once again 480p work the best but the canon color is not right
I think there something in  mlv_rec preview code I need to check in to .

Let me know of any issue as before , I'm mainly want to know if that strange color/pattern show up , it shouldn't  but ......
   

[banertop]

once again, thank you for your work. Amazing things you done for 5d2.

In general, what would be a real benefit of shooting 2.6k, rather than 3k....beside a bit more vertical resolution?

We have now, thanks to you, very nice and stable according to my tests at least, 3k on 5d2.....15 or more seconds, but looks very nice...close to 5d3 max resolution.....less crop then 2.6k....i like it very much......but i think maybe i missed something about 2.6k build, because you guys talk about that build a lot......

Am i missing something?

tnx

[masc]

Thanks for the update. But I think the problems are still the same.

The crop frame is not centered on the sensor. See the picture 2nd row 1st column, this is the 3x3 full sensor. 2nd row 2nd column is the croprec frame. 1st row 1st column I painted the used area.

Then I again get some strange problems with black level: by standard it looks a bit greenish... okay, lower black level in post.
When now scrolling through all the frames, I saw that sometimes some area gets orange blacks... see 2nd row 2nd column carefully, on top it is more orange then bottom. The orange area changes over time.

And last problem: when zooming to 100% I get a kind of bayer pattern in the debayered picture - never saw something like this before... see 1st row 2nd column.

[reddeercity]

I don't fix the centering yet , what to see if that odd color thing was
Still there .

I don't have any issue with debaying , I do not see those problem your reporting.
To me it looks like lens color distortion/sharpness problem.
I use MLVFS ( quick mount ) plus After Effect or MLV Producer
I get correct clean black level every time .

Try it on real scene ( not charts) and see it you get the same result

[masc]

Yes, you also see it in "real world" footage. But with the chart you see it more clearly. With ACR and MLVProducer the black is off too (here uncorrected in the pictures) and the moiree looks even worse:
ACR:

ACR, moiree detail

MLVProducer:

MLVProducer, moiree detail

[Danne]

Check black level auto detection code in raw.c. Or timer tweaks. No code, only guesswork.

[banertop]

i experienced greenish blacks in my tests, to.

but that was with previous build. Did not test the latest one jet.

[reddeercity]

Yes, you also see it in "real world" footage. But with the chart you see it more clearly. With ACR and MLVProducer the black is off too (here uncorrected in the pictures) and the moiree looks even worse:

Interesting , can upload a .mlv file with the problem & also a DNG please ?
The reason is I want to see the debayed image from you so I can compare on my system to see if I'm missing something .
Can you if you have time to take a short 2144x1074 .mlv in 10bit in normal 3xCrop_mode with out the crop_rec enabled , in fact can compare between 10, 12 & 14bit with the crop_rec .
Do you see black level problem in normal 3x3(FHD) 10,12,14bit ? same in normal 3x crop_mode (not crop_rec) 10,12,14bit , is it normal black levels ?

Also does 3008x1080 preset have ok black level on your system ?
Oh yea also you cam setting , preview mode , etc. ....

Edit:
I'll do some more test here and compare
Edit:#2 by chance did you use "Calibrated Colors" ?
If so Don't !!! it screw up the all the raw levels big times , I never use it .

[reddeercity]

@masc -- 2592x1200 , exported dng's from mlv app , mlv_dump & mlvfs(quick mount) on windows7 pro
all have identical black levels & look the same  no bad black level here , can't reproduce you're problematic  results .

No W/B adjustment or any adjustment at all , just pass though from mlv file.
shot at 200 ISO , f4.0 @ 24mm on 24-70mm f2.8L canon EF , w/b 5200k manual
on B/W (Grey Scale) ML Preview Liveview .

MLV App , no correction -- dng export uncompressed


MLV_Dump , 10bit dng export
with this

Code: [Select]

mlv_dump --dng --no-fixcp M10-2118.MLV

MLVFS (Quick Mount) 16bit dng export

[masc]

Note, that not every clip looks like that. In average maybe 2 of 3 are okay. One of 3 has wrong blacklevel and very strange behavoir over time and much more moiree.
Standard 3x3 is always fine, at least I never had a bad clip and I can't see anything wrong there.
Here the frame as MLV, you can use MLVFS to get dng. It is not perfectly focused, it was just a quick test, when I got this. But I think it doesn't matter for your anaysis.
https://www.dropbox.com/s/gwim47xsqdbo65z/M09-1858.MLV?dl=0

My setting was the one you posted in first post in this thread. I don't know where to find "calibrated colors"... so I think I did not use it.

Edit:
Recorded some testclips again, but all fine at 2144x1074 10/12/14 bit or 1856x1044 10/12/14 bit. 2.6K setting @ 14bit kills camera -> battery pull. With todays clips I don't got the green again - last time I got several clips with wrong blacks. Same when I tried your 1st 2.6K module: 20 from 33 clips were fantastic, the rest had "changing blacks over time".

[Igor_Braun]

Hi, my test was okay. I'll be doing tests with high iso.
WB 5400, ISO 1600, the shutter i set to 1/33, it automatically as you said changed to 1/48 (but on the display it showed me 1/24, something like that)



https://drive.google.com/open?id=1awSg_e5ifXgLB2J-xXCxpnJ8Qmo0RVZF

Here the shutter i set was 1/64  (i'm just playing for only test reasons), same as before it changed to 1/48, but i got a big horizontal line, very stylish tho...



https://drive.google.com/open?id=1HJuuYfkMawudk-Wdf8qdaiy8SNglVpEC

Can't wait for the vertical squeezed preset for achieving less cropping by increasing horizontal resolution 

Also, i wanted to say, that with this build i can't format the card from the camera.

[reddeercity]

Note, that not every clip looks like that. In average maybe 2 of 3 are okay. One of 3 has wrong blacklevel and very strange behavoir .......
Here the frame as MLV, you can use MLVFS to get dng. It is not perfectly focused, it was just a quick test, when I got this. But I think it doesn't matter for your anaysis.
https://www.dropbox.com/s/gwim47xsqdbo65z/M09-1858.MLV?dl=0

Thanks for the file ,
before I start to comment on this file , 
To All     This is a big FYI
if ever liveview is messed up of not responding correctly (scrambled , lines or green cast)
You need to refresh liveview (this the cause of 99% of the crop_rec liveview issue reported here)
Press the PLAY Button twice (2 times , 1 to enter to play mode "view a file" 1 to exit)
this will bring you back to 3x3 (FHD) press 5x zoom to re-enter crop_rec.
the odd time you may have too repeat that process a couple a times .

Ok this file was kind of strange , all mlv->dng converters reported different things
It came down to having a corrupted frame , the only convertor that found it was mlvfs(quickmount) .
mlv producer , mlv_dump & mlv app didn't catch it , so this produce wrong white & black levels 118 1013
should be 112(black level) & 1023 (white level)

MLVFS(QuickMount)


MLV_Dump 10bit


The other thing I found to be a problem the shutter speed , the file report 1/40th and with
the preset , I have tuned it to 1/48th if you start at 1/64th at 29.97fps in 3x3 (FHD) .
I did find if you don't use 1/48th shutter this has cause strange liveview behavior .

One other thing that will cause corruption is having overlays enabled while recording raw video
It's far too much load on the cam .

So I don't know your setting but this is what I commend people use to be issue free .
Full manual setting (Important! No Auto setting)
3x3 (FHD) 1/64th 29.97 fps (this will change to 23.976fps @ 1/48th in crop_rec)
ISO 100,200,400 (always refresh liveview after any ISO change)
Do not enabled frame per second  override !!
Do not enable overlays in mlv_rec (default is disabled while recording raw video)
SRM (memory fill rate) "0" seem to be the best on Lexar 1066x cards (optional , shouldn't cause corruption)
Liveview preview , grey scale (B/W) low res.

this part is kind of off topic but anyways .....
The other strange thing was mlv app exported a total pink dng frame ,.



the png frame export was good .
I used ver. 1.6 64bit on windows7 pro  , thou I could see the frame in the preview window fine in the app.
when adjusted the black & white level (112-1023) & unchecked the "don't use camera color matrix " tab it exported a pretty good looking png frame


 

[reddeercity]

Updated the first page post with 2592x1200 test build link

[Danne]

Black level inconsistencies are most probable related to automation. This can be fixed since a1ex put in two options for settling black level. First one needs to know what and how regs were applied.
Seriously, I can't understand your logic with holding open source code that needs to be shared.

[benoit]

Seriously, I can't understand your logic with holding open source code that needs to be shared.

Me too. 6 months of personal coding on one preset with bugs that other people have already see on other plateforms ...
IMHO It's non-sense on an open source forum.
Share is pure power, just think what will be Magic Lantern if a1ex post .mo files with as explanation "i'm busy just read my posts" ...

[masc]

...
You need to refresh liveview (this the cause of 99% of the crop_rec liveview issue reported here)
Press the PLAY Button twice (2 times , 1 to enter to play mode "view a file" 1 to exit)
this will bring you back to 3x3 (FHD) press 5x zoom to re-enter crop_rec.
the odd time you may have too repeat that process a couple a times .

I did.

Ok this file was kind of strange , all mlv->dng converters reported different things
It came down to having a corrupted frame , the only convertor that found it was mlvfs(quickmount) .
mlv producer , mlv_dump & mlv app didn't catch it , so this produce wrong white & black levels 118 1013
should be 112(black level) & 1023 (white level)

Black level is different very often, mostly something between 110 and 118.

...
The other thing I found to be a problem the shutter speed , the file report 1/40th and with
the preset , I have tuned it to 1/48th if you start at 1/64th at 29.97fps in 3x3 (FHD) .
I did find if you don't use 1/48th shutter this has cause strange liveview behavior .

Okay...  I sometimes forget this, but I'll take care in future.

One other thing that will cause corruption is having overlays enabled while recording raw video
It's far too much load on the cam .

Hm... I just had this B/W realtime preview with nothing else shown when recording.

Full manual setting (Important! No Auto setting)
3x3 (FHD) 1/64th 29.97 fps (this will change to 23.976fps @ 1/48th in crop_rec)
ISO 100,200,400 (always refresh liveview after any ISO change)
Do not enabled frame per second  override !!
Do not enable overlays in mlv_rec (default is disabled while recording raw video)
SRM (memory fill rate) "0" seem to be the best on Lexar 1066x cards (optional , shouldn't cause corruption)
Liveview preview , grey scale (B/W) low res.

Shutter... okay... I sometimes forget about, but the other settings should be as you described.

The other strange thing was mlv app exported a total pink dng frame ,.
...
the png frame export was good .
I used ver. 1.6 64bit on windows7 pro  , thou I could see the frame in the preview window fine in the app.

Thanks. I found the bug. It was only when exporting the last frame in a MLV as DNG using "Single Frame Export". "Normal Export" (DNG) was working.

when adjusted the black & white level (112-1023) & unchecked the "don't use camera color matrix " tab it exported a pretty good looking png frame

For me it is still not really good looking. E.g. the blacks get still orange on top. Just the contrast and saturation is a bit lower.

What exactly was changed with the new build?

[reddeercity]

What exactly was changed with the new build?

https://www.magiclantern.fm/forum/index.php?topic=19336.msg214773#msg214773

I'll test record a bunch a clips (20) and see if I can reproduce your odd black & white levels


@masc Ok I can't re-produce your problem , I recorded 13 clips(17.6GB) in a row with high & low contrast .
I recorded until It skipped a frame & stop , then I press record again & so on until I had 13 clips .
I was getting around 400 frames on average before it stopped .

Here are my setting for the test .


So at this point I'm going to say it's a cam setting issue e.g. shutter etc. ...
I'll center the image better & if I have time I'll fix the pinkish canon full color liveview .

I want to move on to finishing coding the other presets e.g. hi frame rate (48fps 3x3 FHD) & 4k anamorphic ,
plus I need to get back on CF card over clocking .

[honza123]

Thumbs up 

[dfort]

Guys -- You should be testing and reporting on the builds that reddeercity has posted so he can continue coding knowing what he has done so far is working:

I want to move on to finishing coding the other presets e.g. hi frame rate (48fps 3x3 FHD) & 4k anamorphic ,
plus I need to get back on CF card over clocking .

Repeating, "Are we there yet?" won't get you anywhere. Believe me!

[Igor_Braun]

Look, actually I think, I can't be useful for Reddeercity more with this new preset, because already posted a small test and told that everything works as described.
But also have problems... But they are of the camera i guess

1. Sometimes there are vertical lines and crazy temperature (WB) changing. There it is:




That's the video:
https://drive.google.com/file/d/1a6PLP2R4e1wLgdBBx8JNwb3OKRDcGV7h/view?usp=sharing

It disappears when I press set button for STOP rolling, and then, when i press REC it appears again. So there are two ways to avoid it: 1. restart the camera 2. wait damn long time watching crazy color changing and vertical lines count increasing, so after a few minutes regularly they are going away.
It happens not only when shooting raw, but I noticed that it took place when the color profile set to cinema style (downloaded from the Technicolor webpage if I remember right)

2. Zoom jumping out.
I had this problem since bought the camera. Already described it here. When I magnify to x5, it jumps out to x1 in most of the cases, so it's truly hard to enable crop rec module and start recording in it.
Guys, maybe you know why is it happening?
That's how it looks from the side (shot on my phone):
https://drive.google.com/file/d/1RSBxPpdYgM38a9qRJfi9IoZbfb3zlUrs/view?usp=sharing

But sometimes everything is ok:

https://drive.google.com/file/d/1CPDy5NCb8kvK4Jhgeh9gWG3VFfBm3K8g/view?usp=sharing (from side)
https://drive.google.com/file/d/1lqBY8uhzRFfAD9oecv_VeNhSvlkJYloO/view?usp=sharing (actual footage)

So, i'm waiting for 4k squeezed preset to compare it with regular 1856 (2.34:1). Just wonder if we really can spot the resolution difference  . And 4k - it's a good way to decrease the crop factor also.

MLV_rec.cfg of that test:
# Config file for module mlv_rec (MLV_REC.MO)

mlv.video.enabled = 1
mlv.res.x = 10
mlv.bpp = 0
mlv.write_speed = 7227