Add m4 to dependency tree

[rouson]

Also bumping the minimum gfortran version to 5.3.0 and setting the default, script-installed gfortran so that it downloads the current 5 branch instead of the current trunk (6.0.0).

[codecov-io]

Current coverage is 38.57%

Merging #103 into master will not affect coverage as of 955d8e3

@@ master #103 diff @@ ====================================== Files 2 2 Stmts 884 884 Branches 0 0 Methods 0 0 ====================================== Hit 341 341 Partial 0 0 Missed 543 543 

Review entire Coverage Diff as of 955d8e3

---

Uncovered Suggestions

1. +1.92% via src/mpi/mpi_caf.c#1892...1908
2. +1.58% via src/mpi/mpi_caf.c#1990...2003
3. +1.58% via src/mpi/mpi_caf.c#1557...1570
4. See 7 more...

Powered by Codecov. Updated on successful CI builds.

[zbeekman]

If you don't mind, let's try an experiment after we merge this branch into master:

Open up a PR with devel as the base branch and master as the compare branch to merge master into devel.

I'm pretty sure most of the time this will be simpler than merging devel into the feature branch and then the feature branch into devel and will ensure that master and devel have a closer common history and should simplify the network graph too. Even if a version number bump gets included in a topic/feature branch with other changes, I think this will just trigger a merge conflict (since devel also bumped the version since forking off of master) but this is easily resolved, and if you turn on rerere git should know how to resolve similar conflicts in the future without asking for your help. At least this is my current working theory 🙏

[zbeekman]

Everything looks good to me. Please take a look at my first comment on this PR before opening up a PR against devel.

[zbeekman]

@rouson So this will always download the latest gcc-5.x stable release? Or does it download the latest 5.x release candidate/bleeding edge?

Are any other download dependencies using the svn fetch method? It might be worth considering other fetch methods for GCC just to remove a dependency (or pseudo/meta dependency) on subversion. Thoughts are:

1. Download a release tarball with wget or curl from one of the GCC mirrors
   • http://ftpmirror.gnu.org/gcc/ will resolve to the geographically closest mirror for downloading
2. The GCC subversion repository is mirrored via git-svn at https://github.com/gcc-mirror/gcc/branches; so if a user doesn't have

This is just food for thought, and this seemed like a decent place to ask you about it.

I'd be happy to add improving some logic surrounding the fetch method to my "someday" todo list.

• Could query the system when first running the script to see if wget, curl or ftp is present, and then where appropriate use one of these download methods.
• I think subversion repos can sometimes be fetched with these tools as well (if they are webdav based) but I don't remember for sure
• Could check in the beginning for svn and git

Any way none of this is urgent at all. If this is of any interest to you we can open up an issue to document and discuss what we would like to see happen.

[rouson]

@zbeekman, the current version of the build script checks out the development version of gcc 5, which presumably means it's downloading a pre-release gcc 5.4. I like the idea of downloading a release from a mirror site instead. That has at least two significant benefits: (1) it makes the installed version and the minimum version consistent and (2) it probably facilitates switching from svn to ftp. My guess is that both svn and ftp are equally widely installed, but the logic of using ftp inside the script is a little simpler -- mainly because it matches the logic of using wget because both download a tar ball. There are several non-obvious special-case branches in the script that are there specifically to deal with the differences between wget and svn. This will eliminate the need for them.

I'll l make that change and then push the resulting commit to the add-m4-to-dependency-tree branch. I suppose it make sense to close this pull request and then create a new pull request that contains the new commit. If you concur, please close it.

[zbeekman]

OK, that sounds like a good plan. If we can remove the reliance on SVN as a fetch method I think that would be a step in the right direction.

As for opening a new PR, I don't think it matters too much, but I would encourage you to just update this one rather than creating a new one. If you add another commit that implements your change to this branch, and then push that commit to the remote branch on origin, then this PR will get updated with the additional commit. I think this will be a good experiment/learning exercise to get to know the features of Github.

$ git checkout add-m4-to-dependency-tree # make sure we're working on the right branch $ # make some changes to implement downloading an official release tarball of GCC from an official mirror $ git add install_prerequisites/build # and any additional files you need to add $ git commit $ git push origin $ # Now the PR #103 on Github.com will include the latest release.

If you feel strongly, however, feel free to close this one and create a new one.

I think using a tagged version of any upstream software is wise since it's possible that a change gets pushed to a development branch could break OpenCoarrays due to a new bug or a regression in the upstream software being introduced, etc. Installing from the install.sh script where uninstalled software is installed from a known version will help us debug user environments.

[rouson]

Ok. I'll try these steps.

One important complication. The logic for using svn needs to stay for use cases in which "build" is a standalone script not invoked by install.sh. I conceived of the "build" script long before "install.sh". The goal of "build" to lower the barrier to people building development versions of gfortran. That barrier is huge and many people (including myself for several years) find building gfortran from scratch to be such a pain that they refuse to do it. With "build", all one has to do is type

./build gcc

and voilá! The gcc trunk gets checked out, built, and installed into the present working directory.

I have to think through how to make the switch cleanly -- probably based on arguments that install.sh is already passing to install.sh. An alternative would be to for "build" to detect whether it was invoked directly by the shell or by install.sh, but that seems dangerous and hard to test.

[zbeekman]

Fair. We can table this change for later too.

[rouson]

I made the change: install.sh now grabs a 5.3.0 release tar ball from a mirror site chosen for no specific reason (ftp.lip6.fr). I don't see where the updated diff shows on this page.

Feel free to merge this latest pull request or I can do it and then attempt to merge master into devel.

[zbeekman]

Well, the latest commit you pushed appears chronologically in the PR discussion. So it is right above your last comment and right below my last comment.

At the very top of the PR, just below the title and above your first comment, and the green "open" icon there are three tabs:

1. Conversation (This tab is currently open if you are reading this)
2. Commits
3. Files changed

If you click that last one, "Files changed", you can see on line 205 the FTP url for GCC 5.3.0

[zbeekman]

Would you mind changing this to http://ftpmirror.gnu.org/ ? (Unless of course you think it's a bad idea...)

From http://www.gnu.org/prep/ftp.en.html :

You can use the generic url http://ftpmirror.gnu.org to automatically choose a nearby and up-to-date mirror.

You can either author a new commit, or amend the latest commit via:

$ # update url in install_prerequisites/build $ git add install_prerequisites/build $ git commit --amend $ git push --force origin

git commit --amend will change the most recent commit to reflect any additional changes you git added and allow you to edit the commit message.

[rouson]

My only concern is whether we can expect the path that comes after "http://ftpmirror.gnu.org/" to be exactly the same. I did some poking around by hand and it seemed that each mirror puts the mirrored file tree in a different path. I suppose the URL you're suggesting must redirect to the right place on each mirror so let's give it a shot.

[zbeekman]

Here is a more accurate URL: http://ftpmirror.gnu.org/gcc/gcc-5.3.0/ for the gcc_url_head

@zbeekman the above one starts with "http". I assume you're not suggesting we go back to wget so I'll check whether it can be accessed via ftp.

[zbeekman]

Every time I click the one I just posted it redirects me to different mirrors, but they have the same contents. This is also the default URL that Homebrew uses for GCC so my confidence in it working is pretty high.

[rouson]

Interesting. I tried editing your comment to see if it would show that I was the one who wrote the new sentence, but it didn't so I'll refrain from that practice going forward.

[rouson]

Wow. After trying a couple of different ways that failed, I now see the ftp can be given a URL that starts with "http". That's some serious magic:

ftp -n http://ftpmirror.gnu.org/gcc/gcc-5.3.0/gcc-5.3.0.tar.bz2

As a bonus, the download appears to be much faster, which I suspect has more to do with the protocol than with the proximity, but who knows...

Who would have thunk it? FTP is a much older protocol than HTTP and HTTP appears to have specific support for redirection, which I'm not sure is the case for FTP. (I've seen some indications that ftp servers can redirect using something called an .htaccess file, but there's not much info about it online.)

[zbeekman]

I haven't tested any of it, but I see no obvious errors or issues.

👍 and if you're comfortable with it, I say 🚢 it!

Just FYI I'm still working on the logic to run Travis-CI tests of the install.sh script. I've gotten it to the point where it builds almost everything it can other than GCC, but I would like to add additional logic so that it only builds MPICH (and maybe it's worth applying this to CMake too...) if the install.sh and subsystems have been modified in the commit range being tested.

Good to know about FTP handling HTTP urls. At some point I may take a whirl at adding some logic that will check if various fetch tools exist (I'm thinking wget, curl, ftp, svn, and git) as well as testing for a network connection (ping http://www.google.com or similar should do the trick) at the beginning of the build script. Then adding some abstraction which will provide a formula to download each missing dependency or software package using the necessary fetch method. For example, GCC 5.3 can be obtained in a number of ways: wget, curl and ftp from standard HTTP and FTP download mirrors, or from a svn tag, or via the git-svn git mirror of the GCC svn repo. Any way, I should probably open a new issue for this. Or better yet it might make sense to break out the build script into its own repo (and then it could be included via git submodule in OpenCoarrays and bundled with release tarballs.

[rouson]

Version number comparisons in install.sh will fail for some common cases. I just finished devising a fix. Later today, I will incorporate the fix into install.sh and push another commit to this branch before merging.

[rouson]

I just tested this commit on Linux with almost all prerequisites missing or below the minimum version number and it appears to work. I had already tested it on OS X with various combinations of prerequisites present or missing. I believe this puppy is ready to merge and I can finally get back to helping to make rain, i.e., the grant proposal. :)

I had to look up "NBD", which is probably a sign that I'm old.

I am very happy to hand off any rebasing to you. :)

Please explain further regarding keeping the history cleaner. By that, do you primarily mean fewer commits or more squashed commits or shorter commit messages? I'll probably continue to commit frequently, but I'd be glad to learn how to squash commits (so much to learn, so little time...). I'm hoping you're ok with the detailed commit messages, but I'm open to influence on that.

[rouson]

This branch is ready to merge into master. If it seems like a good exercise, I can do the rest after you've rebased. I'm guessing the merging of master into devel happens after the rebase.

[rouson]

Or maybe you were referring to the fact that some of the commits don't relate directly to the named purpose of the branch. Or maybe something else... anyway, I look forward to hearing what would represent a cleaner history.

[zbeekman]

The idea of squashing 629e6ea is simply that we changed our minds about a relatively small detail or fixed a relatively small defect that has yet to be introduced to the commit history for master. So rather than having

aaff45 Implement feature xyz
dfaa34 Oops, I should have change pqr in xyz to qrs

in master's history we'll have something more like

adfe32 Implement feature xyz with qrs # combine dfaa23 and aaff45

Interactive rebases let you go back in history and re-order commits, merge commits together (squash them) fixup commits (merge them together, keeping only the first commits log message), edit commits (both contents and log messages) and split commits into two or more individual commits.

If the commit range you're playing with has yet to be pushed to a remote, then I certainly encourage using interactive rebase to cleanup history. However, if you already pushed commits in a range that you're editing with interactive rebase, then the next time you try to push (unless you force it) your push will be rejected. This is because last time I saw the branch in question it looked one way, and I may have started editing it, but then someone else comes along and tells the remote to change everything, so now, at the very least, I am forced to rebase any local commits I made onto the history... any way I suspect I am rambling incoherently at this point enough about this for now.

Note, after I squash the commits, the history on the remote, will be different than your local history, so if you were to, say:

git pull origin add-m4-to-dependency-tree

There would be conflicts and much confusion. However, if you are finished editing the branch, or know you don't have edits in progress, you could then do:

$ git fetch origin $ git checkout add-m4-to-dependency-tree $ git reset --hard origin/add-m4-to-dependency-tree 

you now continue apply commits on top of the rebased branch... any way I'll go rebase this, even though it's not completely necessary, in the interest of keeping a clean history and as a git learning exercise.

[zbeekman]

Also, note how the commits in the PR moved to the bottom (at least until I started commenting on the PR again) since they changed, and comments from the PR diff are now listed as outdate.

[zbeekman]

@rouson it doesn't matter to me whether you or I hit the big green "merge pull request" button here... but I want to investigate a better method of merging these changes into devel after we merge this PR into master, so no need to open a PR against devel

[rouson]

This all sounds pretty complicated. I'll let you hit the green button. I did just realize s straightforward way to handle 2-part version numbers, but I have to pause on development for a few days (hopefully) and here is no reason to wait so merge away.