Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - horshack

Pages: [1] 2
1
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 08, 2020, 08:08:09 AM »
I've created a web-based javascript app that lets you quickly set the camera's clock to +1 day and -1 day to help automate visionrouge's workaround. It only works in browsers that allow you to disable CORS Policy Security. Unfortunately none of the mobile web browsers available support that option, so for now this is limited to home/office/studio use.

Here is the link to the app: http://www.testcams.com/ccapi/datehack.html

Full instructions including how to disable CORS Policy security are in the GitHub repository: https://github.com/horshack-dpreview/canondatehack.html


2
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 04:19:25 PM »
It might be safer to use a fast SD card instead for long 8K and stress test 4K recordings. CFExpress cards do get very hot.

Agreed, SD is likely the better choice for video modes that don't require CFE's speed. It would be nice to see a well-controlled temperature comparison test between SD and CFE for a long-run 8K video (comparing EXIF temps).

3
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 07:27:30 AM »
Okay, different than I was imagining but there might still be a way: phone VPNs to some server, and phone bridges VPN network and wifi network.  Cam connects to phone wifi, server can transparently see all packets from the phone because the networks are bridged.  No NAT required.

The python local solution still seems easier.

iOS's hotspot implementation doesn't support that configuration. If the demand is there the easiest solution is a standalone smartphone app. Canon actually provides the source for a fully-functional proof-of-concept CCAPI Android app. I have no interest in developing for Android though. It's likely a small effort to modify the source. Anyone interested can download the Android app source here: https://developercommunity.usa.canon.com/canon?id=sdk_download. You'll need to register on the site to gain access to the download - registration is free.

4
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 06:57:48 AM »
Oh, direct USB link to the cam from phone?  That's probably what I didn't understand.  In that case you could maybe use the phone as a hotspot, it's VPN'd to some network you control, and then the camera talks to phone via wifi through MITM proxy...  but this is getting ridiculously complicated.

Canon's REST API only runs over its WiFi interface. Using the phone as a hotspot so that an external site could gain access to the camera (ie, one of the existing CORS proxies available online) would require the phone to port-forward unsolicited incoming requests to the camera, the same as a router NAT would. iPhone's don't have NAT functionality for their hotspot feature as far as I'm aware. And if it did it wouldn't require a VPN.

5
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 06:39:31 AM »
VPN allows controlling DNS.  So, you say blah.com is at 192.168.2.2, *that* is a MITM proxy, it requests the real page from camera.xxx, relays that response back to the client *but removes the CORS info*, and then the client doesn't see any CORS, so it does what you want.

The only device is the smartphone running in the field. Where is proxy code running that has IP access to the camera?

6
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 06:31:58 AM »
@horshack - I don't know what you mean by a CORS proxy.  CORS is a server value that is sent to the client by whatever route.  The client can choose to honour CORS policy.  Modifying the client means you can ignore the policy, controlling a proxy in the route means you can remove the CORS policy.  If you have VPN on a client, and camera connected to some machine also connected to the VPN, then client and camera are on the same network.  So I *think* the combination means you can remove the parts that are difficult.  But I've never used Canomate so I could easily be misunderstanding something.

Specifically, Wireguard uses TCP-over-UDP.  UDP for low latency, TCP for convenience.  The two endpoints need a route, they *think* they're talking TCP and are next to each other, but the TCP packets are encapsulated in UDP.  At TCP level the client-server looks one hop away, at UDP level it's whatever route your network / ISP / etc has.

I'm not following you. The client in this case is the browser running my test javascript code issuing REST API requests to the camera's REST API server (CCAPI). There is no way to coerce the browser to accept cross-domain requests unless the HTTP header response from the server contains an Access-Control-Allow-Origin response that specifically allows the domain you're requesting from. The only way around that is either an extension running in the browser that hacks an Access-Control-Allow-Origin field in the response header (doesn't work on iOS because extensions aren't allowed) or by issuing the REST API request through a CORS proxy that issues the request to the intended target on your behalf (camera in this case) and returns a response that adds in the Access-Control-Allow-Origin field. I don't see how running a VPN factors into the equation as a solution.

Canomate doesn't have this issue because it runs as a Python app outside the browser. I tried implementing a REST API client as javascript instead for this specific date/time R5/R6 workaround as a lightweight alternative to running Canomate and also because it would be a portable solution that can run in a Smartphone browser.

7
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 05:03:43 AM »
Yes for sure, it would be good to know when the camera is really overheating based on the sensor and not on the timer.

A poster on EOSHD got 75C (EXIF temp) after 38 minutes of 4K 120fps using @visionrouge.net's workaround. I think that's probably pushing it. It was hot enough to cause the R5's stills temp warning icon to show. He said the CFE was extremely hot when removed.

8
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 07, 2020, 05:00:29 AM »
@horshack - a VPN will look like the same network, you get an additional network interface, with the phone and vpn endpoint being in the same subnet.  Everything is tunneled, so you will see this as one hop, you won't see ISP or phone provider in the middle.  Wireguard or OpenVPN are both free and available on most everything.  Wireguard is supposed to be easier to configure.  You can then configure the VPN to provide DNS, with that DNS server that you control routing whatever names you want to whatever IPs - so you can make the proxy transparent with the only config needed on the phone being the VPN.  Of course, you're trusting the VPN an awful lot!

It looks like your Python works and it's probably easier for people to use.

Thanks. That's probably a bit too much to ask of somebody as a way around the CORS. Btw, how does that create a path between an internet-based CORS proxy and the camera?

9
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 06, 2020, 09:16:49 PM »
True for iPhone, but extensions are allowed on Android.  If you were determined you could also VPN through a MITM proxy to modify the response header to strip CORS, that would work on any phone supporting VPN.  I suspect not a very practical solution for the intended audience :)

Thanks. I can look into Android since I run both iOS and Android. An internet-based MITM proxy wouldn't work since it wont have access to the local phone's IP (unless the port is forwarded through the router and that wont work when away from home network). Running MITM on the phone would work but then you're writing an app anyway to do it. Running MITM on a computer on the local network would work but again not viable when not on home network.

I just tried Canomate with pythonista and it worked on both my iPad and iPhone. Bit of a hassle getting the .py files onto the phone (uploaded them to my Google Drive, then downloaded from Google Drive to Files app, then opened in pythonista). Here's a screenshot from the phone.


10
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 06, 2020, 08:47:43 PM »
CORS is a client side check.  You can disable / fake it with a browser plugin, or a proxy.
https://chrome.google.com/webstore/detail/allow-cors-access-control/lhobafahddgcelffkeicbaginigeejlf?hl=en

Not a great suggestion long-term as it will decrease your security if you forget and leave it active.  Might work for testing.

Not viable for mobile solutions since extensions aren't allowed on those platforms. I'll be testing my Canomate with Pythonista later today.

11
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 06, 2020, 07:48:37 PM »
Here's a Canomate script for automatically changing the camera's clock to +1 day and then back.

Save the contents of the following to a file named DateTimeHack.txt:
Code: [Select]
PrintCameraInfo
SyncDateTime skewsecs=86400
PrintMessageToLog message="Pull battery NOW, wait a few seconds, put battery back in, wait 10 seconds, then press <ENTER>"
WaitForEnterKeyToContinue beep=1
SyncDateTime

To run the script: canomate.py --ip=<camera's IP address> --opfile=DateTimeHack.txt

Output:
Code: [Select]
canomate v0.02 - Automation Utility for Canon cameras (uses Canon Control API via WiFi)
Copyright (c) Horshack, System Time: 09/06/20 10:43:13, Py: 3.8.0, OS: Windows

PrintCameraInfo: Model: Canon EOS RP, S/N: XXXXXXXXXXXX, Firmware: 1.5.0
SyncDateTime: Changed camera time from "Sun, 06 Sep 2020 10:43:13 -0800 DST" to "Mon, 07 Sep 2020 10:43:13 -0800 DST"
PrintMessageToLog: Pull battery NOW, wait a few seconds, put battery back in, wait 10 seconds, then press <ENTER>
WaitForEnterKeyToContinue: Press <Enter> to continue...
SyncDateTime: Changed camera time from "Mon, 07 Sep 2020 10:43:15 -0800 DST" to "Sun, 06 Sep 2020 10:43:16 -0800 DST"
>>>> canomate session over (exit=0), logs at "C:\develop\canomate\canomate-appdata"

I tested this on my Canon RP.

I tried writing a quick JavaScript app that does this in a browser (so we can use portable devices like smartphones) but the Cross-Origin Resource Sharing (CORS) protections prevents the code from performing requests to the camera. Unfortunately the camera is not sending a permissible Access-Control-Allow-Origin in the HTTP header. A developer has already requested a change here: Could a CORS policy be implemented for the CCAPI?. This would have to be written as a Smartphone app instead. Shouldn't be too hard for an enterprising individual  :)

12
Camera-specific discussion / Re: Canon EOS R5 / R6
« on: September 06, 2020, 04:04:59 PM »
Cool hack - it's the clearest proof the "overheating" is timer-based, if you ask me.

Well, this particular hack is easy enough to automate - I wouldn't be surprised if somebody comes up with a one-click version within the next few days or weeks (also mentioned on Twitter). Motivation is high in the R5/R6 user community - much higher than for porting ML on 5D4, EOS R and the like, in my opinion. So, while it's definitely a low-hanging fruit with high rewards, I don't see anyone paying for a slightly more polished version of this hack. A free one will very likely appear sooner or later, without our involvement.

BTW - I feel a lot safer knowing this hack was discovered by third parties - even if it was documented on our forum. There's no single person behind it - all of the previous iterations paved the way to this version of the hack ("standing on the shoulders of giants"), and I'm pretty sure it will be further refined by the community.

Now, back to my (covid-related) research :)

Automation of the time change would be simple - only requires an HTTP GET and PUT over WiFi to "http://<camera ip>:8080/ccapi/ver100/functions/datetime". See my Canomate CCAPI utility at Canomate HomePage and Canomate GitHub Repository for examples on usage. Canomate's SyncDateTime does this.

Automation of the battery pull word require scripting to a network-enabled power switch, like a TP-Link Kasa Smart Plug, connected to a dummy battery inside the camera.

Both could be implemented via a simple Smartphone app.

13
But typical 'engineering' DR calculations using SNR of 1 as the lowest signal don't consider shot noise, correct? Therefore, one could still say that the engineering DR is extended to FWC/read noise @ISO 1600, which is ~14EV. If you're using a lower SNR cutoff of, say, 20, then the DR is much more modest (considering the increase in shot noise contributions you mention).

Right, SNR goes down but if one uses a 1:1 SNR (like DxO does) for DR calculations then then SNR reduction doesn't affect the DR number.

14
Hi horshack-- why would you 'reduce photons collected by half'? The exposure at the sensor plane is not changed; there is only one exposure. The ML hack is simply changing what you do with that exposure data, & so should not affect the SNR save for actually increasing it in shadows by reducing the effects of downstream noise (after ISO amplifier) in the ISO 1600 exposure.

The focal plane exposure (in terms of shutter speed/aperture) remains the same for both ISO 100 & ISO 1600 exposures; it's not like two separate exposure are being made (where the ISO 1600 exposure would have 4 stops less exposure which, yes, would increase the effects of shot noise).

Let me know if I'm misunderstanding you.

For the shadows the ISO 100 lines are being discarded from the composite raw, which reduces the light-capturing surface area of the sensor in half.

15
@horshack,

I don't know who theSuede is (he seems to know some stuff though), but please ask him to read the PDF. I'm pretty sure he didn't.

In deep shadows I only use data from the higher ISO, therefore his averaging formula does not apply here.

Hi Alex, theSuede is a software developer specializing in high-end color workflow products I believe. He is very knowledgeable on most areas of imaging technology but especially on sensors and color. I'm not sure if he's read through your PDF.

I read through the PDF and I saw the section on deep shadows where you only use the ISO 1600 data, indicating "corresponding data from ISO 100 will be just noise". The ISO 100 lines will have more read noise and tossing them out will improve DR but at the same time will reduce photons collected by half, reducing the shot-noise SNR of those shadows from 22.93:1 to 16.21:1, based on some quick calculations assuming the deep shadows start at around 7EV below ISO 100 saturation. As you descend further into the shadows the read noise represents an increasing percentage of total noise. The cross-over point between read noise and shot noise is about -7EV (ISO 12,800 equiv), so even starting in the deep shadows the shot noise represents 50% of the total noise, so discarding the ISO 100 lines is still significant to the total noise.

16
Alex, here is some feedback about the technique from theSuede:
http://www.fredmiranda.com/forum/topic/1227550/0#11686217

17
It's interesting that Canon implemented this capability in their sensor. I believe the technique is patented by Eric Fossum, the inventor of the active-pixel CMOS sensor. Here is one reference to the idea on dpreview; there are other references but I couldn't find them quickly: http://www.dpreview.com/forums/post/32010565

18
Modules Development / Re: [DONE] DotTune AFMA
« on: April 30, 2013, 04:06:18 PM »
I have another suggestion for dot-tune improvement. How about an optional 'cumulative' mode? Specifically, when testing zoom lenses or/and lenses at different distances from the target, one often gets different good dot-tune intervals for MA. E.g, my Sigma 17-50  has MA say -15...-2 at 17mm, and -5...+5 at 50mm. The current dot-tune implementation is not very useful for finding a single compromise value for MA which would be acceptable at any FL. All I get is either one FL or the other FL values; the important interval width information is discarded. But if ML had a cumulative mode, and let me run dot-tune a few times at different FL, and then computed a single compromise value taking into account interval widths, that would be very helpful. In my example, the common MA interval for both FL is -5...-2; the compromise value would be either -4 or -3. At the very least, you could make ML print out the intreval at the end of every calibration; the compromise value would be computed manually.

Generally speaking the width of the confirmed tune range is a function of FL, subject distance and DOF and may not yield any additional information useful for producing a composite tune value for multiple FLs. Experiments would need to be performed to prove otherwise. I'll try to experiment with it next week.

19
Modules Development / Re: [DONE] DotTune AFMA
« on: March 27, 2013, 04:51:34 AM »
I have a 5D Mk II running v2.3.NEXT.2013Mar19 

While running Dot Tune AFMA with 200mm or longer lenses I would sometimes get a bug: Each and every MA box would light up green during the test on all 4 passes. (even with -100..+100 selected). This would result in a new AFMA of "0" and an Error message "Double-check the focus target".  This would happen a few times in a row. Changing Focus Targets, distance, and lighting, had no effect. I would then exit the program, reboot the camera, and try various other things until it went away.  This only happened a few times and I am not even sure how I was able to correct it.

Any thoughts on what was going on?

Your symptom sounds like the AF tune value being scaled by the ML logic wasn't being applied/utilized by the camera's AF logic for some reason. Please report back if you're able to reproduce this.

20
Modules Development / Re: DotTune AFMA
« on: March 04, 2013, 11:00:26 PM »
Alex, here is logic that can make the algo parameters tunable if you'd like to go that route. I couldn't get a submenu-within-a-submenu working so these options are at the same menu level as the others:

top of module:
static CONFIG_INT( "dottune.number_scans", dottune_number_scans, 2);
static CONFIG_INT( "dottune.max_time_10ms_initial_af_confirm", dottune_max_time_10ms_initial_af_confirm, 200/10);
static CONFIG_INT( "dottune.min_time_10ms_sustain_af_confirm", dottune_min_time_10ms_sustain_af_confirm, 500/10);


// we have to wait until the AFMA change is operated into Canon firmware
static int afma_ack = 0;


afma_auto_tune():
    SW1(1,100);

    for (int k = 0; k < dottune_number_scans; k++)
    {
        for (int c = 0; c <= 80; c++) // -20 ... 20 and back to -20
        {
            int i = (c <= 40) ? c-20 : 60-c;
            set_afma(i * range_expand_factor, afma_mode);
            msleep(100);
           
            int fc;
            // initial focus must occur within quick period to exclude AF confirm lags
            fc = wait_for_focus_confirmation_val(dottune_max_time_10ms_initial_af_confirm*10, 1);
           
            if (fc)
            {
                // weak or strong confirmation? use a higher score if strong
                // focus must sustain for long period be considered strong
                if (!wait_for_focus_confirmation_val(dottune_min_time_10ms_sustain_af_confirm*10, 0))
                    // focus sustained
                    fc = 3;
            }           
           
            score[i+20] += fc;
            afma_print_status(score, range_expand_factor);
           
            if (!HALFSHUTTER_PRESSED)
            {
                beep();
                set_afma(afma0, afma_mode);
                NotifyBox(2000, "Canceled by user.");
                return;
            }
        }
    }
    SW1(0,100);


Bottom of module:
static MENU_UPDATE_FUNC(dottune_max_initial_af_confirm)
{
    MENU_SET_VALUE("%d ms", dottune_max_time_10ms_initial_af_confirm * 10);
}
static MENU_UPDATE_FUNC(dottune_min_sustain_af_confirm)
{
    MENU_SET_VALUE("%d ms", dottune_min_time_10ms_sustain_af_confirm * 10);
}

static struct menu_entry afma_menu[] = {


And added menu options:
            {
                .name = "AFMA mode",
                .priv = &afma_mode_index,
                .select = afma_mode_toggle,
                .min = 0,
                #ifdef CONFIG_AFMA_WIDE_TELE
                .max = 3,
                #else
                .max = 2,
                #endif
                .choices = CHOICES(
                    "Disabled",
                    "All lenses",
                    #ifdef CONFIG_AFMA_WIDE_TELE
                    "This lens, wide/prime",
                    "This lens, tele"
                    #else
                    "This lens"
                    #endif
                ),
                .help  = "Where to apply the AFMA adjustment.",
                .icon_type = IT_DICE_OFF,
            },
            {
                .name = "No. range scans",
                .priv = &dottune_number_scans,
                .min = 1,
                .max = 99,
                .help  = "Larger values increase sample size.",
            },
            {
                .name = "Max 1st AF confirm(ms)",
                .priv = &dottune_max_time_10ms_initial_af_confirm,
                .min = 10/10,       /* 10ms */
                .max = 10000/10,    /* 10 seconds */
                .update = dottune_max_initial_af_confirm,
                .help  = "Smaller values are more discerning.",
            },
            {
                .name = "Min cont AF confirm(ms)",
                .priv = &dottune_min_time_10ms_sustain_af_confirm,
                .min = 10/10,       /* 10ms */
                .max = 30000/10,    /* 30 seconds */
                .update = dottune_min_sustain_af_confirm,
                .help  = "Larger values are more discerning.",
            },
            MENU_EOL,
        },
    },
};

21
Modules Development / Re: DotTune AFMA
« on: March 04, 2013, 07:57:44 PM »
Another benefit of exposing the raw parameters is that it can help with debugging any body/lens-specific ML DotTune issues.

22
Modules Development / Re: DotTune AFMA
« on: March 04, 2013, 06:39:18 PM »
Thanks for the detailed response! But this begs a suggestion - would it be possible to let the user choose both the number of passes and the duration of one measurement, from the ML menu?
I've considered that, exposing the low-level algorithm parameters to the user in the menu. It would be up to Alex. There would actually be three parameters:

1) Total number of passes
2) Maximum period allowed (ms) before first AF confirmation must occur. This is to catch delayed confirmations that should be excluded. This is currently hard-coded at 200ms
3) Minimum period (ms) that AF confirmation must sustain without any dropouts. This is to catch multiple confirmations that should be excluded. This is currently hard-coded at 500ms.

Instead of exposing these parameters it might be more user-friendly to define a "Fast" vs "Slow" option, the former using the current thresholds and the latter using expanded thresholds.

23
Modules Development / Re: DotTune AFMA
« on: March 04, 2013, 06:24:14 PM »
I just did fairly comprehensive testing of dot-tune/ML on my 50D, using  magiclantern-v2.3.NEXT.2013Mar02.60D.550D.600D.50D.500D.5D2.1100D.zip. The results are reported here:

http://photography-on-the.net/forum/showthread.php?t=1280584

In particular,

Thanks for your observations, they're greatly appreciated.

Regarding ML's DotTune and how it may sometimes miss the multiple confirmations  for the outlier tune value values, there are two possible strategies at work here, one which is used by the manual DotTune process and one used by ML's DotTune.

In the manual process the user is instructed to hold AF for 5 seconds, which catches > 90% of the multiple AF confirmations that should be excluded from the tuning range. This strategy is important because the manual techique is based around a decisive determination of indivudal tune values in a "single pass".

The second strategy, which is used by ML, is to decrease the sampling duration but increase the sampling frequency. Instead of waiting 5 seconds to detect multiple confirmations at each tune value, ML waits for only 500ms (1/2 second). But ML does 4 "sweeps" (passes), which means 4 500ms samples, for a total of 2.5 seconds of sampling. Multiple-confirmations occur at irregular intervals into an AF hold, so a tune value may produce multi-confirmations at 3 seconds into the AF hold on one sample but at only 300ms into another. In practice it roughly works out to about the same, and because ML's DotTune uses a weighted algorithm for the final tune value determination, it will properly demote tune values which had one or more multiple-confirmation samples relative to values which had no multiple-confirmation samples. That's not to say ML will not sometimes fail to exclude a tune value that a thorough manual DotTune would otherwise catch. There's always room for improvement. It's a balance between expediency and accuracy. ML DotTune could increase the sampling duration to catch maybe 30-50% more multiple-confirmations per sweep, but with 4 samples it wont affect the final result in the majority of DotTune cases.

24
Modules Development / Re: DotTune AFMA
« on: March 04, 2013, 05:18:12 PM »
But is "all lenses" scaled across focal length too? Lots of little quirks it seems.
Not sure, but I would think yes (for zooms).

Another curious question is how Canon's firmware handles a per-lens wide/tele value of 0. The 5DM3 manual says firmware will scale for zooms even if only the wide or tele tune value is set...so how does their firmware distinguish a user-specific tune value of 0 vs a non-set (unassigned) tune value, which is also represented as zero in the menu? Would seem they would need an assigned/unassigned bit somewhere, otherwise the firmware would always scale on the "non-configured" end to 0, which should be quite different than scaling when both the wide/tele are set to identical values. Based on these uncertainties I would strongly suggest owners to always tune both the wide and tele ends.

25
Modules Development / Re: DotTune AFMA
« on: March 04, 2013, 04:03:59 AM »
I have a 100mm macro lens that I use both for macro shots and for portraits. Running DotTune at both a macro and a portrait distance gives different results (-5 at portrait, +4 at macro). Sound I use the average? 0 and -1 lies in the acceptable range for both. Does that sound reasonable or am I missing something? The author suggests 50 x focal length. I almost never use this lens at 5m so it doesn't seem appropriate to set the MA at that distance.

It's not unsual for lenses to exhibit focus-distance variance in tune values. You can either choose the value for a single distance if that's what you shoot at most often, or use the midpoint of the distance values if that produces acceptable results for you.

Pages: [1] 2