Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - dfort

Pages: [1] 2 3 ... 92
1
Camera-specific discussion / Re: Canon 100D / SL1
« on: Today at 06:53:58 PM »
@theBilalFakhouri detailed how he did it but even if you just put the camera on a tripod and shoot it off your computer screen it should work.

Every video mode at every bit rate and compression setting would be best but right now what I'm most interested in are mv1080crop and 5x zoom -- the 1:1 sampling video modes using 8...11bit, 12bit and 14bit lossless compression.

Don't bother testing Dual ISO with the test pattern. We know there is a problem but it shouldn't affect the focus pixels.

2
Camera-specific discussion / Re: Canon 7D Mark I
« on: Today at 05:41:45 PM »
Does IO crypt also work with the new ML?

I have never used that module but if I got the addresses right it should work.

https://bitbucket.org/hudson/magic-lantern/pull-requests/878/update-to-7d206/diff#chg-modules/io_crypt/io_crypt.c

[EDIT] The example shows the addresses for 2.0.3 so I updated the example. The values used in the module didn't change with the firmware update.

3
Camera-specific discussion / Re: Canon 100D / SL1
« on: Today at 05:36:22 PM »
Sorry, I meant that all of the shots were underexposed. You're far from clipping the highlights.



This is what the Dual ISO histogram looks like before processing:



I'm not an expert at exposing Dual ISO but this looks underexposed to me.

Focus pixel fixing seems to be working, though maybe the 100D doesn't need the 1-pixel offset pass that the 700D needed. What would really help is shooting that test pattern @theBilalFakhouri used. It is still available on his Google Drive from the link in his post.

The main issue here is that cr2hdr isn't processing your 8...12bit Dual ISO shot.

4
Camera-specific discussion / Re: Canon 100D / SL1
« on: Today at 04:42:24 AM »
Quick report.

I can't get the 8...12bit Dual ISO isn't working with MLVFS or mlv_dump/cr2hdr. The normal shots worked fine but are quite underexposed.



Looks like maybe the 2-pass map isn't needed on the 100D mv1080crop 8...12bit? Hard to tell with this test because there are so few focus pixels showing up.



Here's the 14bit lossless Dual ISO after playing around with it. Recovered lots of shadow detail but it is a bit noisy. No focus pixels, though.



Well, almost. There were a few along the edges that persisted.



By the way, earlier I was getting lots of focus pixels. When testing out MLVFS I often pull out the fpm files to see the focus pixels. Guess what? I forgot to put them back.  ::)

All fine now!

5
The latest focus pixel map files are here:

https://bitbucket.org/daniel_fort/ml-focus-pixels/src/8bf385a7e828aa07aa605bc911ea94921b5c3c58/focus_pixel_map_files/?at=default

As you discovered there is a lot of duplication in the map files. Read the focus pixel topic if you’re curious why that is.


Sent from my iPhone using Tapatalk

6
Camera-specific discussion / Re: ML on EOS-M2
« on: Yesterday at 07:05:54 AM »
Got it working on Windows Subsystem for Linux (WSL) -- (Scream like a little girl!)



(does this make any more sense now?)

Yes it does. Anytime the ROM is patched the firmware signature changes. Is that the lesson here?

Didn't test on Mac though; I'll try later if it still doesn't work. Without loading ML, is the Canon menu still working?

Yeah, looks like there is a Mac issue, but only with patches.gdb. Running ML with debugmsg.gdb does show the firmware signature and it matches what I'm getting in camera. It was just that the commit you pointed out was on patches.gdb so I thought it was necessary to run it with patches.gdb.

One other observation - on Mac it pauses for user input while on WSL it doesn't:

Code: [Select]
---Type <return> to continue, or q <return> to quit---
I merged in the new-dryos-task-hooks a while back but maybe I screwed it up. I'll try it again and see if I can track down the bogus stubs.

..a tiny hack in menu.c to bring menu in QEMU (not needed for camera), and if there are any buttons not recognized, just print their codes.

A tiny hint on how to print out the button codes?

7
Got a suggestion for the documentation.

After hours of struggling to get QEMU working again on Windows Subsystem for Linux (WSL) after a bunch of Microsoft and Ubuntu updates -- what is really important but not obvious is having to start the X server. BTW, VcXsrv isn't working here after the updates but Xming is and it is just as easy to install and run.

[EDIT] Oh yeah, don't forget this:

Code: [Select]
export DISPLAY=:0

8
Camera-specific discussion / Re: Canon 100D / SL1
« on: January 16, 2018, 07:21:29 PM »
Glad to hear about the absence of focus pixels though I tried different apps keep seeing them. Maybe I need to take a vacation. The video I posed was done on Switch. It usually does a good job on Dual ISO.

Looks like your latest has focus pixels and the Dual ISO pattern.



The focus pixel map file for the 100D in mv1080crop mode is one of the most complicated with 206,400 mapped pixels covered in two passes. Here is what the pattern looks like:



On the 650D/700D/EOSM I had to create four pass map files to get it working with 8...12bit lossless mv1080crop.

9
Camera-specific discussion / Re: Canon 100D / SL1
« on: January 16, 2018, 05:28:49 PM »
Besides the cr2hdr issue -- I see dots!


@IDA_ML - Could you do a short test using the same settings but without Dual ISO?

10
Camera-specific discussion / Re: Canon 100D / SL1
« on: January 16, 2018, 02:19:52 AM »
Looks pretty good over here. How are you processing?

This is with MLVFS and Adobe Camera Raw.





Do your blinds have that pattern? I don't think that's a Dual ISO pattern. [EDIT] I think therefore I'm wrong.

This is what it looks before the cr2hdr process:



I don't see any focus pixels anywhere on this shot.

[EDIT] I do see some issues on the other two shots. Need to take a closer look at what is going on.

11
Camera-specific discussion / Re: ML on EOS-M2
« on: January 16, 2018, 01:15:23 AM »
@a1ex committed a change for the EOSM2 on the qemu branch so of course I tried it out. Wish I could say everything was rainbows and unicorns but--

Code: [Select]
./run_canon_fw.sh EOSM2 -s -S &
arm-none-eabi-gdb -x EOSM2/patches.gdb -ex quit

...
Breakpoint 1, 0xff0c5144 in ?? ()
Patching LeoLens (infinite loop)
[MPU] Received: 06 05 09 11 01 00  (PROP_LV_DISPSIZE - spell #22)
[MPU] Received: 12 11 09 15 00 00 00 00 00 00 00 00 00 00 00 00 00 00  (PROP 80050020 - spell #23)
[MPU] Received: 14 13 09 25 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  (unnamed - spell #24)
[MPU] Received: 06 05 01 5a 00 00  (PROP_CONTINUOUS_AF_VALID - spell #25)
[MPU] Sending : 06 05 01 59 01 00  (PROP_MOVIE_SERVO_AF)
[MPU] Received: 06 05 09 2f 01 00  (unnamed - spell #26)
[MPU] Received: 06 05 0e 22 1e 00  (unknown - unnamed)
    70:   121.344 [MR] mvrChangeAckCBR : Video - Mode=0, Type=2, Rate=30, GOP=15
    88:   125.440 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050038
    89:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050041
    90:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050042
    91:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050043
    92:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050044
    93:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105004F
    94:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050050
    95:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050051
    96:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050052
    97:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105010E
    98:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105010F
    99:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050110
   100:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050111
   101:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0104000B
   102:    60.160 [LVCOM] InitializeLiveViewDefectDetection
   103:    60.416 [LVCOM] ExecuteDefectMarge Start
   104:    60.416 [LVCOM] ExecuteDefectMarge End
   106:    61.184 [LV]  PROP_LV_BLOCK  PROP_LV_UNBLOCKING 0
   108:    62.208 [AUDIO] RegisterCBRSDIOCableConnect
   109:    64.256 [WFT] InitializeAdapterControl END
SD: Unknown CMD1
[SDIO] Error
SD: Unknown CMD1
[SDIO] Error
SD: Unknown CMD1
[SDIO] Error
   124:   151.808 [SD] ERROR SDINTREP=0x00000000
   125:   151.808 [SD] ERROR UNEXPECTED ERROR
   131:   113.152 [RSC] WARN ILLEGAL SUBFREECLUSTERCOUNT ShootStorage.c 686
   157:   184.832 [DISP] ERROR PROP_TFT_SETTING_PARAM DON'T FIND
   158:   184.832 [DISP] ERROR Factory Adjust For TftCom
   159:   185.088 [MR_MOV] (Empty Func) MVW_RegisterXmpDataCallback
   160:   142.336 [TCH]Touch IC Ver : 0x0000
   161:   569.600 [TCH]ERROR  TouchPanelIC Ack Error
   162:   569.856 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   163:   570.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   164:   570.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   165:   570.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   166:   570.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   167:   570.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   168:   570.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   169:   570.624 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   170:   570.624 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   171:   572.672 [IMPP] H264E InitializeH264EncodeFor1080pDZoom
   172:   572.672 [IMPP] H264E InitializeH264EncodeFor1080p25fpsDZoom
   179:   578.816 [STARTUP] startupInitializeComplete
   185:   580.864 [CEC]CEC OFF

I usually use debugmsg.gdb instead of patches.gdb so I gave that a go but couldn't get Hello World to print, much less the firmware signature. Going back to my previous build it worked--sort of. As expected, it printed the wrong firmware signature.

There is some positive news to report. Although I still can't get to the ML menu on the camera or QEMU I hacked the ML/SETTINGS/MAGIC.CFG file to see if any of the features are working--say like focus peaking?

It works! Yay!

So what's the next step of this walk through? Seems like we need to find the code for the Trash button, which on this camera is also the down key or bottom of the thumb wheel. Maybe do a serial flash dump? I could probably get everything set up but how to trigger a module without access to the ML menu?

12
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: January 16, 2018, 12:10:26 AM »
@dfort: the M2 signature issue was fixed; check c33141cd12a9 and the (updated) guide.

Great. I'll try it out and report on the EOSM2 topic. BTW--great progress on QEMU. This last build went pretty much on autopilot.

13
Camera-specific discussion / Re: Canon 100D / SL1
« on: January 15, 2018, 06:44:36 PM »
I see lots of problems on one of your shots. Left over focus pixels and a pattern over light flat areas (not to mention too many refrigerator magnets):



Could you post trimmed down MLV's of your 100D tests so we can take a closer look?

Code: [Select]
-- MLV output --
  -f frames           frames to save. e.g. '12' saves frames 0 to 12, '12-40' saves frames 12 to 40. forces --no-audio switch

Example--save the first 4 frames to output.mlv.

Code: [Select]
mlv_dump -f 4 [input.mlv] -o [output.mlv]

14
Camera-specific discussion / Re: Canon 7D Mark I
« on: January 15, 2018, 02:58:05 AM »
Good news for 2.0.6 -- I've got a working ML-SETUP.FIR Been testing it out and it is working fine.

So now you don't need to downgrade to 2.0.3 in order to set the camera bootflag.

As usual, my test builds are posted on my Bitbucket downloads page.

15
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: January 14, 2018, 10:40:48 PM »
...we should update the signature in src/fw-signature.h

Yes, the same thing happened to me on the EOSM2. AFAIK QEMU skips the firmware signature so it will still work.

16
Made some more changes on the fpm.sh script.

When in doubt whether to use mv720 or crop_rec map files, default to crop_rec because it will work with either video mode.

Using the -n option to create full raw buffer sized maps will also default to crop_rec because it is assumed that these map files will be used in MLVFS that can't can't yet differentiate between crop_rec and mv720.

I also looked into removing dependency of the Crop metadata but it turns out that it is necessary to line up the map files properly.

Still todo -- run that 1-pixel offset map for mv1080crop mode only if it is 8...12bit lossless compression. Not a high priority because running additional passes that aren't really needed doesn't seem to hurt the image. At least not much.

17
The latest map files are in the ML Focus Pixels Bitbucket repository. Just replace the ones that are in MLVFS with these.

18
Hey @theBilalFakhouri - @bouncyball updated mlv_dump on steroids with the latest focus pixel maps so going back to your shot, here it is processed through Adobe Camera Raw maxing out on the highlight recovery, clarity, vibrance and saturation. That should bring out any remaining focus pixels.

mlv_dump --dng --white-fix=4931


There is an area that still shows some focus pixel artifacts:



I doubt they would be a problem unless you stretch the image as far as I have. Adding chroma smoothing eliminates it and I don't see any other issues introduced by chroma smoothing.

mlv_dump --dng --white-fix=4931 --cs3x3


Click on the images to see them full size.

I take it you didn't use a VAF filter on your 700D. There are a few post production tricks you can use to get rid of the aliasing.

Let's see what happens.


Hum--is there a problem with the youtube tags on the forum? Here's the link: https://youtu.be/x3uZzVivlzY

Yep, some aliasing issues but that has nothing to do with the focus pixels.

19
My first success at removing focus pixels was to use the chroma smoothing in raw2dng. In fact I setup my first ML development environment specifically to get this feature by uncommenting just this one line:

Code: [Select]
/* useful to clean pink dots, may also help with color aliasing, but it's best turned off if you don't have these problems */
//~ #define CHROMA_SMOOTH

I sometimes still use chroma smoothing to get rid of some stubborn focus pixels. I recently discovered a similar feature in dcraw with the '-m' option. So I thought I'd see how well these methods work without using the focus pixel map files and it looks like I found the same bug in mlv_dump, MLVFS and dcraw.

Let's start with dcraw. The you can look at the code on dcoffin's website -- search for the median_filter() function to see how it works.

I ran these tests with mlv_dump and MLVFS by removing the focus pixel map files and only using the chroma smoothing options with the same results.

This is what it looks like using mlv_dump on steroids with the --no-fixfp option -- thanks for this @bouncyball



Note that there's a row of focus pixels right on the bottom edge of the image. Now let's do two passes with the median filter '-m 2'



Much better but you can still see them. I found that '-m 4' was the limit, anything more didn't help and I tried it all the way to '-m 100'



Note that the focus pixels on the bottom just won't go away.

Now let's try it again only this time using mlv_dump with some chroma smoothing. I got the same results using cs2x2, cs3x3 and cs5x5:



With dcraw we could shave off one pixel from the bottom of the image but with mlv_dump we would have to remove the bottom five pixels. In addition, there is another problem with some focus pixels that were close to the side of the image.

cs2x2 and cs3x3 gave the same results:



cs5x5 also showed a green focus pixel:



Is there anything that can be done to improve how chroma smoothing works on the edges of the frame? The chroma smoothing code is a bit over my head.

20
Splitting multi-pass map files into separate image files is great for debugging (dedotting?) problem files. Put each of the map files on separate Photoshop layers and you can figure which focus pixels each pass is hitting.

I've got another idea, one that goes all the way back to the first post on this topic:

Here is what I ended up doing to get rid of just one focus pixel--and the diagonal secondary pixel.

Code: [Select]
1247 121 0
1248 120 0
1248 121 0
1248 125 0
1248 126 0
1248 122 0
1248 123 0
1248 124 0
1248 126 0
1248 122 0
1248 120 0
1248 124 0

Note that I hit the diagonal pixel just once but had to work top and bottom to middle and hit the same pixels several times in different sequences. Think of it as stippling a brush to spot a print--in photography school I spent many hours spotting prints. (By the way, this a map of all the focus pixels for the 650D that was used on the PinkDotRemover tool.) [EDIT] Broken Link

Here is the result. I circled the area where the hot pixel was located and yes, it looked just like those other ones before running the dng through dcraw.


Hang in there, this is going to be a little abstract. When we used to spot prints using Spotone it wasn't just one hit of the brush on the print and you're done. You need to hit the same spot repeatedly in order to blend it in to the surrounding area and match the grain pattern. In other words you need to stipple out the spots. Let's say that we have a hot pixel that is ruining all of our footage. How about making a "Spotone" pass by simply adding those coordinates at the end of the fpm file? That last pass shouldn't be sorted and in effect it is doing many passes over the same area until the problem pixel disappears. This is a little hard to do manually but in a GUI each time you hit the brush on a pixel, it should build the map file in the order the pixels were selected. That would simulate the process of spotting a print.

Does this make sense?

In effect it might be doing what the dcraw -m option does but only in the areas that are needed in a predetermined order.

21
raw_twk is causing problems on the EOSM possibly because it has maxed out on the number of threads it can handle.

This might also explain why things were working better on my older 7D builds before I added the raw_twk module.

As far as the future of the raw_twk module, I think that the plan might be to eventually incorporate something more efficient in mlv_play to playback lossless compressed files. Then again if we can get H.264 proxy working you will be able to playback the H.264 file with audio in camera.

Off topic: @IDA_ML -- There has been lots of work recently to get the focus pixel map files working for 8...12-bit lossless compressed files in MLVFS. Most of the testing was done on EOSM and 700D but it could really use a thorough test from an experienced 100D user.  ;)

22
When I started this topic I thought it would be challenging but at some point the focus pixel issues would be completely resolved. There seems to be no end in sight.

I've been running acid tests on the new focus pixel maps with all the problem MLV files I could find and they seem to be working fine with MLVFS. Here's an oldie that @DeafEyeJedi shot a while back.

100D mv1080 Dual ISO before focus pixel fixing


No need to pixel peep to find those focus pixels! Looks pretty good after fixing the focus pixels.



However, I wouldn't recommend Dual ISO in movie files because of the aliasing issues. Check out the far bleachers and the grid lines on the field.


If you want HDR video check out the work @Danne is doing on the HDR script keep original framerate topic.

Hum--wonder if the Dual ISO aliasing can be smoothed out with the methods discussed in these topics:

Experiment - reducing aliasing in post using optical flow

Reducing aliasing in post using enfuse/hugin align_image_stack

23
What is "outside"?

I'm using fpmutil to look at the multipass map files and I might be starting to understand a little bit of why these "phantom" focus pixels are showing up in dcraw. I'm not really familiar with the way dcraw works thought it is certainly no secret. It looks like the focus pixels (bad_pixels in dcraw) are fixed in the same order that we are creating the map files--starting from the top left (0,0 position) and reading left to right one line at a time. No big surprise there but are the fixes happening before or after debayering? I'm not entirely sure but isn't part of the process to blend surrounding pixels? If that's true then reading from left to right, top to bottom, the surrounding pixels are blended into this pixel:



Thus leaving behind a "phantom" pixel that wasn't on the original image:



This can be resolved using a multi-pass map file. That pixel is processed after the row below it so the area sampled is not contaminated with pink. Makes sense but what about this pixel?



Using a multi-pass map that top row is processed last so why isn't it picking up the pink? Is it because adjacent focus pixels are saved in complimentary colors that cancel each other out? IDK.



The mystery remains unsolved.

Quote from: Old Joke
I keep seeing spots before my eyes.
Have you seen a doctor?
No, just spots.

Continuing with the dcraw tests, there was still one last row of those "phantom" pixels near the bottom of the frame. On MLVFS @dmilligan is doing a special process on the focus pixels near the edge of the frame. I have been recommending adding some chroma smoothing to get those last few stubborn focus pixels. I couldn't find a chroma smoothing option in dcraw but it does have this:

Quote from: dcraw manpage
-m number_of_passes
        After interpolation, clean up color artifacts by repeatedly applying a 3x3 median filter to the R-G and B-G channels.

Running two passes (-m 2) cleaned up those phantom pixels. At one point we were running chroma smoothing to eliminate focus pixels so I was wondering if maybe this can be used the same way in dcraw. Just to see what would happen I ran it ten times (-m 10) and it worked! Well, almost. There were still focus pixels showing up on the edges of the frame. Some of those pixels weren't even in the original so let's just say they were very bright "phantom" pixels.

I'm getting a lot of mileage on this 14-bit lossless MLV with the shifted pixels. I've been throwing more and more coordinates into the map file wondering just how far we can take it before it breaks. This is what the 4-pass mv1080crop focus pixel map looks like:



It doesn't seem to be hurting dcraw:



MLVFS looks like it might have a problem with it.



So I thought I'd try the old single pass map file. Along the way I discovered something interesting--recall this:

"Normal" EOSM mv1080crop 1600x640
Code: [Select]
    Crop: 176x222
     Pan: 172x223

"Broken" EOSM 14bit lossless mv1080crop 1600x640
Code: [Select]
    Crop: 176x220
     Pan: 170x220

Shifting the old map file 2-pixels did the trick.



However, the results look pretty much the same as the mega map file. Maybe even a little worse? I'm pretty sure I haven't mixed them up.



It seems logical that we should be mapping only the fewest locations where focus pixels are showing up. However, it doesn't seem to be harming the image if we map all possible locations. This is good news for the map files we're making for MLVFS because we don't need to swap out map files--I believe Switch is doing this for the crop_rec map files.

While I was working on this post @bouncyball has been pushing updates so the testing continues!

24
Quick update -- doing a second pass on the row of pixels that dcraw left behind worked.

Before


After


This isn't needed on the MLVFS maps but they should probably be updated so everything is consistant.

Of course the more testing the better, especially on the 100D.

25
It seems like a bug in panning mode.

The terminology could get confusing. As you know the panning information we're talking about here is the position of the upper left corner of the image to the full raw buffer. When I wrote the script I wanted to cover all the possible ways it could be used including legacy RAW and DNG which don't have that metadata. What the script does is to assume the image is centered on the buffer:

Code: [Select]
./fpm.sh -vm mv1080crop EOSM_14bit_losless_broken_raw_buffer_000000.dng

Camera:               EOSM
Camera Model:         80000331
Resolution:           1600x640
Video Mode:           mv1080crop
Full Raw Buffer Size: 1872x10**
Pan:                  170x221
Crop:                 176x220
Output File:          EOSM_mv1080crop_1600x640.txt
File Format:          dcraw

Not exactly the same as the original MLV but surprisingly it worked.

They could be secondary artifacts after interpolation of adjacent or nearly adjacent pixels. That is why we are using 2pass mode for some flooded by focus dots maps. 'dcraw' does it differently.

That's exactly what is going on. It was probably happening for a while but we haven't noticed it until now. Let's take a really close look at the problem pixels.

No focus pixel map


With focus pixel map


Left over focus pixel after processing in dcraw


What threw me off was that the leftover pixels were not on the original but they were being covered by the map file


My theory is that dcraw is interpolating all the surrounding pixels including that pixel that doesn't need fixing. We can't just remove them from the map file because focus pixels can show up in these locations. Why is it happening on just that row and not on other adjacent pixels? I don't really know but I'll bet that if that row is processed on a second pass it will work in dcraw.

Of course we're already doing two passes on this so maybe it needs third and fourth passes? It is just a matter of how the coordinates are sorted on the map file.

Pages: [1] 2 3 ... 92