Nightly builds page redesign (static HTML)

Started by a1ex, September 20, 2016, 06:47:25 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

a1ex

Jenkins is overloading the server too much for my taste lately, so I'm considering rewriting the nightly builds page as static HTML, without any JavaScript. Another reason for the rewrite: the builds page is impossible to load on slow network connections.

Any volunteers to help me with this task? I'm going to use a Python script similar to this one, and here's what I came up with so far (edited manually, based on the previous template):

http://builds.magiclantern.fm/550D.html

Here's a proof of concept Python code to retrieve Jenkins build data, using JenkinsAPI:

from jenkinsapi.jenkins import Jenkins
J = Jenkins('http://builds.magiclantern.fm/jenkins')
B = J['500D.111'].get_last_good_build()
artifact = list(B.get_artifacts())[0]
print artifact.url


Feedback is also welcome (while I'm at it). There's some extra functionality I'd like to add, too.

magnesiumnitrate

Any idea how long the nightly server will be down? I just bought  600D and want to instal. If it's going to be a while I'll try and figure out how to compile but otherwise I can just wait. Thanks!

a1ex

No idea, it all depends on the volunteers. The previous configuration can no longer be used because of a hardware issue on the server, unfortunately.

drbohlen

Hi, I went to the download page today to download the latest version of ML for the 60D and found that the download site is down.  I had a previous version on my card a while ago, but since then my wife has formatted my card and I lost magic lantern.  I haven't really needed it so I didn't worry about it, but I'm filming a wedding this weekend and needed to put it back on.  I might be able to go back and search my computer for the old magic lantern, but it was literally over a year ago that I had it on.  I don't know how the download site works, but is there any way someone could send me or put on dropbox or some other download site the latest stable or nightly of ML for the 60D?  I would greatly appreciate it.  Thanks.

tecgen

> Any volunteers to help me with this task?

I help you a1ex.
Canon 5D Mark II, 50D, 550D/Rebel T2i, EF 40mm f/2.8 STM, Sigma 18-35mm f/1.8, EF 85 f1.8, EF 135 f2.8 SF, Zoom H2n

axelcine

EOS RP, 5dIII.113/Batt.grip, 5dIII.123, 700d/Batt.Grip/VF4 viewfinder + a truckload of new and older Canon L, Sigma and Tamron glass

macdabby

I would recommend running jenkins just once every 24 hours or something to that effect. Or if the repo has webhooks (like github does) then you can have it fire a rebuild request to your build server whenever new code is merged. From there, you could do something as simple as letting your webserver list directory contents of a build folder. You can do this with apache or nginx. If you wanted a better interface than that, I'd be happy to volunteer for that. I have a platform that I use called Lightning for dynamic content. It's an extremely lightweight CMS built for speed and high volume requests. It can even run without a database, although even with it, you can handle plenty of requests. If you want to talk about it send me a PM.

edgewood58


tecgen

A1ex, I've got a simple Python script running now, that reads a generic template file, takes the URL of the latest nightly build from Jenkins and write everything into a new HTML file. I could not manage to read anything else (e.g. build time, changeset hash or the number of downloads) from the Jenkins instance so far. I need to understand the Python JenkinsAPI better if it's even possible. To bad currently Jenkins is not reachable here. So I couldn't test out things.

https://bitbucket.org/tecgen/magic-lantern/branch/nightly-builds-page-redesign-static-html#diff

> There's some extra functionality I'd like to add, too.
Could you tell me what you like to add in future?

Do you plan to use one Python script that generates multiple HTML file or do you need to call it with parameters for each camera model?


Canon 5D Mark II, 50D, 550D/Rebel T2i, EF 40mm f/2.8 STM, Sigma 18-35mm f/1.8, EF 85 f1.8, EF 135 f2.8 SF, Zoom H2n

a1ex

Quote from: macdabby on September 21, 2016, 07:32:43 PM
I would recommend running jenkins just once every 24 hours or something to that effect.

Yes, that's my plan. The download links appear to require Jenkins running though, but this should be fixable.

Before Jenkins, we had some directory listing (h5ai) and a cron job that ran daily. It did the job well, but a nicer interface is tempting as well.

Quote from: tecgen on September 21, 2016, 09:24:38 PM
A1ex, I've got a simple Python script running now, that reads a generic template file, takes the URL of the latest nightly build from Jenkins and write everything into a new HTML file.

Sounds good!

Quote
Too bad currently Jenkins is not reachable here. So I couldn't test out things.

Just restarted it, hope it's fine now. It used to lock up within minutes after starting, but since I've disabled the builds page, it's a little better. Not sure how to diagnose it - the symptom is very high CPU usage with no apparent reason (and no obvious increase in website traffic - it's actually pretty quiet).

Quote
Do you plan to use one Python script that generates multiple HTML file or do you need to call it with parameters for each camera model?

I don't mind, either way should be fine.

Quote
> There's some extra functionality I'd like to add, too.
Could you tell me what you like to add in future?

Sure, most of it should be obvious from the extra menu items from the example HTML, but I'll give some details below.

Here's what I'd like to address:

* design: OK to keep it along the same lines as the old one?

* compatibility with Windows or Mac browsers: does my HTML template look fine there?

* validation: I'm not good at web development, so other than the w3c validator, I'm not sure what to look for.

* top menu:
  - main builds: ports merged to mainline
  - early ports: ports that are still in progress and can be compiled from their own branches (e.g. 1200D)
  - additional modules:
    - here I'm thinking to list the modules that are not widely used, or still experimental (e.g. io_crypt, tiltcalc, raw_diag)
    - I can compile from user repos as well
    - todo: a list with what's worth including here
    - what about naming this just Modules, moving the modules section from the features page here, and listing additional modules below?
  - utilities
    - things like mlv_dump, raw2dng, cr2hdr, with both win/mac builds
    - todo: a list with what's worth including here
  - experiments
    - builds from branches like new_sound_system, crop_rec, iso-research
    - what about compiling pull requests automatically, so users can give feedback easier?
  - other stuff worth including?

* buttons:
  - change log
     - this used to be a popup box, which looks nice, but requires javascript, which I'd rather avoid
     - I'm open to suggestions on how to make them more obvious
     - what about displaying the changes for latest build right below the buttons?
     - the changes for old builds could be in a large HTML page; maybe only show the last month or so?
  - build log: plain text is fine for me; better suggestions?
  - old builds: large HTML page?
  - screenshots and tests:
     - proof of concept here
     - we may start with some menu screenshots from qemu (should already work on some cameras like 60D, 5D3, 500D, 1200D), and do the rest later.
     - a simple test would be to start each binary in QEMU and check whether ML config file gets created (this should also work on models with SD card that cannot boot the GUI yet)
     - this section has huge potential IMO (automated burn-in tests and such), but let's start with something easy, just to get the ball rolling.
  - discussion: just links to the forum (for each camera)

We don't need to finish all of these in order to get it back online; just the basics (previous functionality). I'm just thinking in advance, what would be nice to have over the next few months.

That's it for now, thanks for helping on any of those items. I expect to have some more time next week, to work on it; meanwhile, I'm mostly available during evenings (European time). We may discuss on IRC as well if you prefer.

Ottoga

@Alex

Suggested additional top menu items "Popular Apps", "Canon Firmware Files"

The "Popular Apps" menu could be populated with links to the home pages of 3rd party applications used by members (Public Domain only and/or Commercial).

E.G.:  AutoStakkert, CombineZP, DeepSkyStacker, Astro Photography Tool, Startrails, Startrax, Raw Therapee, UFRaw etc...

The "Canon Firmware Files" menu item would be populated with links to ML supported Canon firmware files by camera model.

Suggested items for the "Utilities" menu could be links to things like:

  • Card testing and undelete utilities that have been recommended by community members.
  • EOSCard
  • EOSinfo - shutter count program
  • Magic Rescue
  • dcraw
  • exiftool

Maybe make a distinction between pure utilities like card testers vs workflow apps. e.g.: I would treat the likes of BarracudaGUI, CeroNoice, MLVProducer, etc. as workflow apps

Under "Additional Modules" suggest to include: "bolt_raw", "bulb_nd" and "foc_inf"
EOS 7D.203, EFS 55-250mm, EF 75-300 III, Tamron 16-300 DiII VC PZD Macro, SpeedLite 580EX II.

tecgen

A1ex, I make some progress with the Python script. Now I could generate HTML files with the data from Jenkins, that contain almost every information, like your 550D.html page example in your first post, but for every camera currently supported by ML. Missing within the HTML is currently the hash of the Mercurial changeset from the repository, the commit messages and the number of downloads. I need to extract the first two values from a structure which appears to me like a JSON object or array.


[
{
'node': '15eb9aaa05c022c637c6ece48095a219ee7069c3',
'deletedPaths': [],
'modifiedPaths': ['modules/raw_rec/Makefile'],
'author':
{
'absoluteUrl': 'https://builds.magiclantern.fm/jenkins/user/alex@thinkpad',
'fullName': 'alex@thinkpad'
},
'timestamp': -1,
'rev': 12166,
'commitId': None,
'date': '1473959453.0-10800',
'merge': False,
'addedPaths': [],
'_class': 'hudson.plugins.mercurial.MercurialChangeSet',
'affectedPaths': ['modules/raw_rec/Makefile'],
'msg': 'raw2dng and related: use HOST_CFLAGS (should fix building raw_rec with older compilers)'
}
]


The number of downloads must be read somehow with the JenkinsAPI, but I don't know how to do it yet. I pushed my latest progress into my development branch.
https://bitbucket.org/tecgen/magic-lantern/commits/c562c7bfd54ffc088ce79d3aba9c8b0349be3970?at=unified
Canon 5D Mark II, 50D, 550D/Rebel T2i, EF 40mm f/2.8 STM, Sigma 18-35mm f/1.8, EF 85 f1.8, EF 135 f2.8 SF, Zoom H2n

jzedurqn

Hey folks,

Before you finish a non-Javascript version. I created a Javascript-based website for people to use in the meantime.
https://coloraggio.github.io/builds.magiclantern.fm/

It is as fast as Jenkins API allows (I optimized query parameters to make responses faster).
I also implemented a client-side caching to save useless request when user is navigating back and forth between pages.

GutterPump


DeafEyeJedi

Indeed that's a nice look to start off with @jzedurqn!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

a1ex

@jzedurqn, thanks for the facelift, I like it!

@tecgen: the number of downloads is not given directly by Jenkins - that's my hack, and the info is available in a separate JSON file ( http://builds.magiclantern.fm/download-stats/api/json ). It only contains the most downloaded ones, and it's a static file (recomputed periodically).

BTW, the old site is available at http://builds.magiclantern.fm/index-js.html, for reference.

reddeercity

Yes looks very good , but I notice you can only download the latest nightly builds , What about older builds ?
How will there be accessed ? As we all know the latest nighty builds sometime can very problematic .
So we my have to reverting back to a older build.
Before you could show all the older builds from a dropdown menu , I think this needs to be re-implemented .

a1ex

Current status: https://builds.magiclantern.fm/5D2-212.html

(a small HTML page with latest build, a larger HTML page with all the old builds, and a huge HTML listing all the changes as well)

The changeset info from Jenkins appears both broken (e.g. incorrect on 1100D) and incomplete (missing branch info, for example), so I guess I should take just each build's changeset from Jenkins and lookup the other changes directly on mercurial.

I'm also thinking to show only the large changes (e.g. merged pull request XYZ, with link), since a large page with all that stuff doesn't look quite easy to read. Better ideas?

aedipuss

a1ex - the page design looks nice, it follows with what has gone before it.  lean but usable. :)
canon 600d
canon 60d - Nightly.2016Apr17.60D111
canon 70d - 112

anDyIII

Hi folks,
I tested the new builds page a bit and it looks good! Great work!
I just have a few things to suggest for a better readability.

1st. It would be nice to group the changeset list and make it like the Older builds one. So you can click on "changes", or whatever you want, and show the list and then hide it.
2nd. Could you set the the fw version under the camera model a little bit bigger? My eyes are still good but I had to read it two times to choose the right version.  :D
Another possible solution for this, that would also shorten the list, would be: Write the camera model as simple text and write one or more firmware versions as links on its right or below it.
For example:

5D Mark III  1.1.3  1.2.3

5D Mark III
1.1.3 1.2.3

Keep up the good work guys!
5D Mark III
www.facebook.com/groups/magiclanternitalia/
www.andreacasanova.eu

Greg

Quote from: anDyIII on September 24, 2016, 04:02:41 PM
2nd. Could you set the the fw version under the camera model a little bit bigger? My eyes are still good but I had to read it two times to choose the right version.  :D

html :
<span>Canon Firmware Version:<span>1.1.3</span></span>

css:
.camera-list span span {
    display: inline;
    font-size: 12px;
    font-weight: bold;
}

anDyIII

@Greg

Thanks.

I updated my post right now with another solution I thought that could also clean up the cameras list.
5D Mark III
www.facebook.com/groups/magiclanternitalia/
www.andreacasanova.eu

a1ex

It's live!

Credits go to tecgen (initial script), jzedurqn (layout ideas) and Greg (CSS help). Thanks, guys!

There's still some missing functionality, and contributions are still welcome. Tidying the HTML, fine-tuning for mobile devices, checking the links, suggesting useful information... you name it.

garry23

A1ex

Had a quick look and seems pretty good.

One observation is that I would suggest we need a way for seeing what each new build adds/subtracts from the previous build, ie info/text about functionality changes or enhancements.

At the moment 'all' you have is a link to the code.

Thanks to you and others for your continuing dedicated work.

Cheers

Garry

a1ex

Click on "Older Builds", then on "show details".

Any suggestion about how to make this more readable / obvious?