ProcessTwoInTwoOutLosslessPath

[dfort]

For simple silent, maybe one of these is the solution?

src/raw.c

Code: [Select]

#if defined(CONFIG_5D3) || defined(CONFIG_100D)
    (*height)--;
#endif

#if defined(CONFIG_700D) || defined(CONFIG_650D)
    (*height)++;
#endif

Not sure which direction to go but I'm betting on subtracting instead of adding. I'll post two new test builds for the 70D. This shouldn't affect FRSP.

[andy kh]

all i did was in video mode not in photo mode. im not at home at the moment. i wil do all the test 2moro morning as soon as i get back home and report

[reddeercity]

@a1ex

Right; mind running a "hg bisect" to find out when the black level changed? This trick is very useful to track all sorts of bugs, where you know it worked in some older version, but some reason is no longer working in recent ones and you have no idea where to start looking.
You could start with something like:

Code: [Select]

cd platform/5D2.212
hg bisect --reset
hg up fa84643 -C # some old changeset from 2013; still compiles cleanly here
make clean; make
# make sure the black level is about 1024 with this build
hg bisect --good

hg up unified -C # current unified
make clean; make
# make sure the black level is about 1728 in the same mode
hg bisect --bad

# test intermediate builds suggested by hg, and answer with --good or --bad
# you will need to test ~ 12 builds to find the changeset that resulted in different black level


I run the bisect , but I may have messed up , (first time trying this )  when i run the "good bisect" nothing really came up .
On running the "bad bisect" after it did it's thing i was left with a message

The first bad revision is:
changeset:   8123:ce6210d55f62
branch:      unified
user:        Audionut <***********@*****.com> (I blanked the email address)
date:        Fri Aug 23 04:21:39 2013 +0000
summary:     Bunch more feature.html links

Is this what you are looking for , did I do it right ?
so changeset : 8123:ce6210d55f62 on the unified on 8/23/2013 , is the problem ?
should I compile ?  if so (I never had to compile different changeset before , 1st time)
in the unified branch , something like

Code: [Select]

hg up unified changeset : 8123:ce6210d55f62
I'll try it and see what happens (I'll need to learn this sooner then later)

just to confirm what I did for the "bad bisect"
"hg up unified -C # current unified
make clean; make
# make sure the black level is about 1728 in the same mode
hg bisect --bad"
copy this on command line window on 5d2.212 platform and run it

[reddeercity]

Compiled the change set that was marked as bad , Could only run silent.mo mlv_rec.mo would not load , on the is old core just error out .
took 2 silent photo (simple) 1880x1250 , got two black levels (1791 & 1792) white level (11130) on both images
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/87190001.DNG
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/87190002.DNG
 

Code: [Select]

Magic Lantern v2.3.NEXT.2018Mar08.5D2212
Mercurial changeset: ce6210d55f62 (unified)
Built on 2018-03-08 00:57:33 by ml@ml-pc
Camera  : 5D2
Firmware: 212   -e   -e

[dfort]

That can't be right. Commit ce6210d is just a change to feature.html links.

I was thinking that this also applies to the 7D so I thought I'd run the same bisect test.

Code: [Select]

hg bisect --reset
hg up fa84643 -C
First issue was that the old changeset was using an old ARM toolchain so I had to install that. Then there were several things in debug.c that wouldn't compile so I kept throwing out functions until it compiled. Whoa--that is an old one. This was before modules? Shot a simple silent still and checked it in exiftool:

Code: [Select]

Black Level                     : 2047
White Level                     : 15000

Hum--this is way off the 1024 we're looking for on the 5D2. Looked in raw.c and saw that RAW_DEBUG_BLACK was in this build so I defined it along with RAW_DEBUG and this time the console popped up and showed black=2055 and white=15000 on the console. Ok, close enough.

Code: [Select]

hg bisect --good
hg up unified -C
@reddeercity - The "-C" option discards all the changes so all that stuff I threw out didn't mess up that changeset.

Turned on the black debug options again and whoa again--



Lots of information but where's the black level? Ok, shot a simple silent still and checked it in exiftool:

Code: [Select]

Black Level                     : 2047
White Level                     : 13947

The light conditions are changing rapidly where I'm at right now--watching the sunset. So some change is probably expected but it looks like this exercise doesn't apply to the 7D?

@reddeercity - If I were to continue on this and in fact the unified build is showing a black level of "about 1728" in your tests the next step would be:

Code: [Select]

hg bisect --bad
You should see in your terminal:

Code: [Select]

Testing changeset 11285:eb053c87b498 (4859 changesets remaining, ~12 tests)
543 files updated, 0 files merged, 319 files removed, 0 files unresolved
What hg bisect just did was to divide the changes between the two changesets (4859 changes) and update to the first changeset to test. I prefer doing this with two terminal windows, one for hg bisect and the other for building and installing ML. That way I can scroll back through the hg bisect list and see what is being tested. Of course the next thing to do is to test this changeset in the camera, mark it good or bad and hg bisect will set up the next changeset to test. Keep going and eventually it will find where things went bad.

[reddeercity]

great thanks , yes I got that testing 12 builds etc. ... I kept pressing enter key(windows) and after the 6th there was no more change sets that are bad and then "The first bad revision is"
message come up . Now I think about it , I bet I should have been testing the changeset after each won came up . I run it again and see what happen .

Yes it's the old "stable build" . I was playing around with that old core got 10-12bit code implemented have some good success , even had a option to set bit rate to 16 in the old .raw format.
I abandoned it to work on UHD/compressed raw on the new ml backend , I did learn a lot from it .

[andy kh]

samples from dfort build
FRSP https://expirebox.com/download/77b127ef8cfab41b14aa3087cd6679ee.html

movie mode canon menu set to 1080P
simple lossless https://expirebox.com/download/d7379ccf0ec72b64e1086db455749b68.html

simple uncompress  DNG https://expirebox.com/download/1e66883a3eff9faac1b2341fa7fdbc4a.html

canon menu set to 720p: 1)Simple lossless https://expirebox.com/download/4c4738e227abf5fb3e78357c2118e114.html
                                     2) https://expirebox.com/download/1b45d686a6a6595be3dcba505aaa699c.html
                                     if i shoot something shining objets like lights or laptop screen, images are not black

i wil upload files from esas"s build in the evening

[reddeercity]

Came across some a useful windows app called https://www.impulseadventure.com/photo/jpeg-snoop.html
supports the following file format

Code: [Select]

•.JPG - JPEG Still Photo
•.THM - Thumbnail for RAW Photo / Movie Files
•.AVI* - AVI Movies (*mjpeg)
•.DNG - Digital Negative RAW Photo
•.PSD - Adobe Photoshop files
•.CRW, .CR2, .NEF, .ORF, .PEF - RAW Photo
•.MOV* - QuickTime Movies, QTVR (Virtual Reality / 360 Panoramic)
•.PDF - Adobe PDF Documents
Very good diagnostic tool  , not that I know everything about .
I using to check compression on the CR2 files and there's a lot of  info , so much there I never know there all that data 
What I discovered was there is 3 jpg images in the cr2
there's the thumb nail image ,
Full Res Jpeg (8bit YCbCr4:2:2) with picture profile applied

Code: [Select]

_MG_8718.CR2.export.000002.jpg
File Size                       : 1443 kB
File Modification Date/Time     : 2018:03:06 21:15:01-07:00
File Access Date/Time           : 2018:03:07 22:19:20-07:00
File Creation Date/Time         : 2018:03:07 22:19:20-07:00
File Permissions                : rw-rw-rw-
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Image Width                     : 5616
Image Height                    : 3744
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:2 (2 1)
Image Size                      : 5616x3744
Megapixels                      : 21.0
and the raw Jpg 

Code: [Select]

_MG_8718.CR2.export.000003.jpg
File Size                       : 21 MB
File Modification Date/Time     : 2018:03:06 21:15:01-07:00
File Access Date/Time           : 2018:03:07 00:25:24-07:00
File Creation Date/Time         : 2018:03:07 00:25:24-07:00
File Permissions                : rw-rw-rw-
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Image Width                     : 1448
Image Height                    : 3804
Encoding Process                : Lossless, Huffman coding
Bits Per Sample                 : 14
Color Components                : 4
Image Size                      : 1448x3804
Megapixels                      : 5.5I extracted the jpg from the CR2 with jpeg snoop.exe  , here is the full _MG_8718.CR2.log.txt  some very interesting thing of note
Clipping data

Code: [Select]

*** NOTE: YCC Clipped. MCU=(   2,   0) YCC=(   -1,  127,  128) Y Underflow @ Offset 0x0000E87C.4
*** NOTE: YCC Clipped. MCU=(   2,   0) YCC=(   -1,  127,  128) Y Underflow @ Offset 0x0000E87C.4
*** NOTE: YCC Clipped. MCU=(   3,   0) YCC=(   -1,  127,  127) Y Underflow @ Offset 0x0000E87C.4
etc. ......
*** NOTE: YCC Clipped. MCU=(   9,   0) YCC=(   -1,  127,  127) Y Underflow @ Offset 0x0000E87C.4

Code: [Select]

Compression stats:
    Compression Ratio:  3.98:1
    Bits per pixel:     6.03:14:1 compression , interesting .

Code: [Select]

  RGB histogram in DC (before clip):
    R  component histo: [min=   -4 max=  256 avg=   93.5]
    G  component histo: [min=   -1 max=  255 avg=   81.7]
    B  component histo: [min=   -3 max=  253 avg=   62.0]
  RGB clipping in DC:
    R  component: [<0=  732] [>255=    1]
    G  component: [<0=    2] [>255=    0]
    B  component: [<0=  855] [>255=    0]
    RGB histogram in DC (after clip):
    R  component histo: [min=    0 max=  255 avg=   93.6]
    G  component histo: [min=    0 max=  255 avg=   81.7]
    B  component histo: [min=    0 max=  253 avg=   62.0]
Average Pixel Luminance (Y):
    Y=[ 82] (range: 0..255)
Brightest Pixel Search:
    YCC=[ 1018,  -26,   11] RGB=[255,255,247] @ MCU[  4,  1]etc. .....

The reason I did this was to investigate the (5D2) full res size as different app are reporting are not the same.
from 5616x3744 to 5634x3753(UFRaw) with the same CR2 , strange

So is the cr2 just being crop in certain App like photoshop etc. ... but really is 5634x3753 just like the FRSP ? if so then it shouldn't be a problem save lossless at the res.

One last thing when I loaded the cr2 on ExifTool , it give me a BOAT load of info , specially about the compression   Exiftool_MG_8718.CR2.txt
e.g.

Code: [Select]

Raw Image Segmentation  : 2 1936 1920and this

Code: [Select]

Sensor Width                    : 5792
Sensor Height                   : 3804
Sensor Left Border              : 168
Sensor Top Border               : 56
Sensor Right Border             : 5783
Sensor Bottom Border            : 3799Black level

Code: [Select]

Average Black Level             : 1024 1024 1024 1024
Raw Measured RGGB               : 108619 154380 158979 55163
Per Channel Black Level         : 1024 1024 1024 1023
Normal White Level              : 14977
Specular White Level            : 15489One nice  thing about it is there a full res jpeg at 1.4 mb , would be great to be able save  that jpeg stream to image sequence *.jpg (dreaming right now)
cr2 reports black level as 1024 so , I need to find out more about the 1791 level .
Edit: the 3 exported jpg files from the cr2 with JpegSnoop
Thumbnail
Full Res 5616x3744 jpeg
Raw-Lossless, Huffman coding
original CR2

[Danne]

I noticed too the beefy sized jpg preview in CR2 files. They are the same size in sRAW as well. There is also a tif and a smaller jpg thumb in there. I use this preview in HDR workflows. You can fetch the preview with exiv2:

Code: [Select]

exiv2 -ep3 -l . *.{cr2,CR2}

[dfort]

@reddeercity - If you aren't testing each changeset how do you know whether to mark it good or bad?

I'm a little lost over what is going on with this exercise. I tried compiling that old changeset and it built fine on the 5D2 while it was a struggle on the 7D. Is there an issue with the current unified code on the 5D2? What do the "good" and "bad" silent stills DNG files look like? The ones you posted look fine to me. Everything coming out of the 7D is looking good on all the changesets I tried. Of course none of the Digic 4 cameras can do lossless compression even though we're pretty sure we found the right addresses to do this. That's what we're trying to figure out, right?

@andy kh - Ok, the simple silent isn't completely black. That's a step in the right direction. Which build worked better, height++ or height-- ?



Of course the goal is to get it to match the uncompressed as closely as possible:



BTW - that esas build is still here: https://bitbucket.org/es_as/magic-lantern/downloads/

As for the jpg previews in CR2 files--isn't that what the image playback is using? Don't see much point in having to do in-camera decoding unless you're manipulating the images in the camera. Maybe that's why we can't find the lossless decoding in the 5D2?

[reddeercity]

@reddeercity - If you aren't testing each changeset how do you know whether to mark it good or bad?

I mess up on the what I needed to do my bad , I'll redo it tomorrow

I'm a little lost over what is going on with this exercise. I tried compiling that old changeset and it built fine on the 5D2 while it was a struggle on the 7D. Is there an issue with the current unified code on the 5D2? What do the "good" and "bad" silent stills DNG files look like?

From my understanding we are seeing where the black level changed from 1023 in (2013) to 1024 & 1791 current from from this post here

mind running a "hg bisect" to find out when the black level changed? This trick is very useful to track all sorts of bugs, where you know it worked in some older version, but some reason is no longer working in recent ones and you have no idea where to start looking.

The bad is  either pink Or green cast (black level problem)

The ones you posted look fine to me. Everything coming out of the 7D is looking good on all the changesets I tried. Of course none of the Digic 4 cameras can do lossless compression even though we're pretty sure we found the right addresses to do this. That's what we're trying to figure out, right?

Yes

As for the jpg previews in CR2 files--isn't that what the image playback is using? ....  Maybe that's why we can't find the lossless decoding in the 5D2?

Maybe  , There's more going on with that JPCore then we know . I wonder if on D4's it's looking to make or compile all those 3images + the History gram thumbnail
run jpeg snoop and you will see what I mean, thought the TwoIN-TwoOut  , as it's noted in the code we are only use 2 channels (one in one out) if I read the code right  say something to the effect
"we don't use these" I have a feeling it something with the driver out .

[a1ex]

The bad is  either pink Or green cast (black level problem)

I'm not talking about black level issues, I'm talking about black level value ("good" = near 1024, "bad" = near 1792). This is just a convention - "hg bisect" only accepts two labels - good or bad. You can assign them otherwise if you prefer. The image will be good in both cases, as long as the black level metadata is correct. If it's not correct, you can adjust it, but your answer ("good" or "bad") should not depend on whether the metadata is correct or not. The answer should only depend on the black level value that gives correct image (or --skip if, for some reason, the test could not be run with some particular changeset).

If in doubt, check the optical black area from a (simple) silent DNG. This feature was implemented before raw recording, and wasn't changed as often as raw video, so it's more likely to get a successful test run with silent pictures, rather than raw video. Again, ignore the metadata if the image is bad; only consider the mean (or median) value of the optical black area (or simply the black level value that gives good image).

If all else fails, try reading a tutorial.

[andy kh]

i dont remember. i wil test esas"s build now and retest your build and let you know

[andy kh]

samples of esas's build

samples simple lossless 1. https://expirebox.com/download/a5d97c7b46b79a01c319d49849a65a1d.html

2.  https://expirebox.com/download/e6f035fdc9f19ce6ac4b57136842af9e.html

simple dng   https://expirebox.com/download/66e0dde45f502f0194eb3a81a59506da.html

FRSP  https://expirebox.com/download/8601c7616b80cbc76100a8d51fd40388.html

[dfort]

Great -- while you were testing I made yet another build. What happens if we include all the stuff we did with the 6D/650D to fix the problems we were having on those cameras? This probably won't work but it might be worth a try. The build is called 70D_crop_rec_4kitchen_sink and is on my downloads page.

[andy kh]

height-- is better
samples of height-- https://expirebox.com/download/77aa034865de95b3286991c66f88cf1f.html
https://expirebox.com/download/3f43abd05c91d067066639f72534574f.html

samples of height++  https://expirebox.com/download/a4c8f9cf605d703f3c7b126cf2ed3138.html

https://expirebox.com/download/860dc9b62e066ff02b63c04ea0d86452.html

[andy kh]

crop rec 4k kitchen  https://expirebox.com/download/ccf6a9ad466f7e22385c0123c3da6f20.html

https://expirebox.com/download/5db28f39dfadf77202a779bb937d8d35.html

[dfort]

Well, that experiment blew up. Still it is good to know that those settings aren't the ones causing the issues.

I'm a little confused, some of your simple silent stills are uncompressed yet they are showing problems. Is simple silent on the 70D build posted on the nightly downloads page also having this issue?

[andy kh]

im not sure about issues
i have a sample from the nightly build
simple dng  https://expirebox.com/download/3cc3fcf483cc7692b6963afc8d80b097.html

[dfort]

That DNG from the nightly build looks good.

Maybe we should back up to and see if there's anything else that we need to resolve the issues on the 70D:

Uncompressed full-res silent DNG, please (ideally the full set of VRAM dumps).

For reference: 70D photo capture log from nikfreak.

Here's a few things to track down the green/pink issues:

Re white level:

http://www.magiclantern.fm/forum/index.php?topic=16040.msg191131#msg191131
http://www.magiclantern.fm/forum/index.php?topic=16040.msg190644#msg190644
https://bitbucket.org/hudson/magic-lantern/commits/98539896ebd056e0a2128658aacd73b757a78b08
https://bitbucket.org/hudson/magic-lantern/commits/ab792b7c5137fceee4016bf9ef88767d031d27c5

These are the current 70D tweaks:

Code: [Select]

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }


[a1ex]

Added 6D and 650D lossless support to the main crop_rec_4k branch (with minor differences from dfort's version, hopefully non-functional for these models). Feel free to compile from source until the builds will appear.

70D is included too, but doesn't work well yet. Can be fixed by somebody who has the camera and some tinkering time in their hands.

Thanks dfort and other contributors!


BTW, noticed the lossless encoder doesn't really care about W*H being modulo 16 - removed this restriction and got a few extra pixels. Please double-check. I'd expect 1736x696 in 720p on 18MPix models.

[dfort]

Great! Killed my old PR's and took down the test builds to reduce clutter.

I'll keep working on the 70D but need user input.

[saulbass]

Just had a quick play with magiclantern-crop_rec_4k.2018Mar10.650D104 - on my 650D.

Looks great - it's amazing what you've managed to squeeze out of this camera.

One tiny caveat, the crop rec res in 720p & 1080p is 1920x1076. Is there a reason why we couldn't get 1080 vertical? dforts earlier builds managed a full 1920x1080.(i.e. crop_rec_4k.2018Mar05.650D104)

Anyway - major progress - haven't had time to look at anything else much - FYI the non crop rec res remains at 1736x976 as per your last builds. (i.e. crop_rec_4k.2018Mar05.650D104)

Compressed recording looks fine btw.

[a1ex]

I assume you are talking about the x5 zoom mode. This change is the culprit. From the samples I have, there are 28 pixels at the top, just like in all other modes; previous builds used to skip 26 pixels in zoom mode, so the first two lines were black. Your DNG sample shows them.

The other two lines at the bottom seem valid. Sorry, missed this one.

In movie crop mode (not to be confused with x5 zoom or crop_rec), skip_bottom was declared as 2 in previous builds. I reduced the height by 2 (for consistency with other models - height-- vs height++) and redefined skip_bottom as 0 - that should not change the active area (compared to previous build in this mode).

However, looking again at the EOS M sample I have from this mode, there are no bad lines at the bottom (height=1060 aka height++); there are just 28 black lines at the top (so there may be 2 additional lines of usable data, similar to x5 zoom). So, I don't know what was the reason for using skip_bottom = 2 for mv1080crop - were things different on other models?

The non-cropped modes had declared 4 skipped lines at the bottom. That was the reason for changing it, as 5D3 and 100D had some issues with extended resolution (they required height--, autodetected height was too much and caused hiccups / skipped frames, but 700D/650D/M used height++). I did not see a valid reason to read two additional lines (compared to other models) and then skip 4 at the bottom.

720p: there were 2 bad lines at H=726, so there must have been 3 at H=727. Reduced height by 2 -> 725 (for consistency with other models) and declared 1 skipped line (hopefully squeezed one extra good line, compared to dfort builds).
1080p: there were 2 bad lines at the bottom with H=1190 (but ML was skipping 4); reduced H by 2 for consistency and declared skip_bottom = 0 (hopefully squeezed 2 additional lines, compared to dfort builds).

Please double-check the resolution in 720p and 1080p (to make sure the additional lines are there, with valid image data). I've only counted the pixels from VRAM samples; did not test these settings on a camera.

BTW, I don't have the following VRAM samples:
- 650D movie crop mode (MVC-1080), but I have one from EOS M
- 700D (other than scattered files picked from the forum, I don't have a full set of VRAM dumps)
- 100D (same as above)
- 70D (same as above)
- 60D, 5D2, 500D, 550D, 1200D (lower priority; I have a few dumps, but not the full set)

[saulbass]

@a1ex - thanks for the reply. I'm back stacked with work for a few days but when I get a chance will look into these tests.