add new aarch64 linux nodes

[futurejones]

Part 3 of restructuring the aarch64 server resources.

Add new build nodes for -

• Amazon Linux 2
• CentOS 8
• Ubuntu 18.04
• Ubuntu 20.04

I have setup a docker hub repo with pre built images for these OS's to simplify the setup.
https://hub.docker.com/u/swiftarm
Details of Dockerfiles used are here -
https://github.com/futurejones/swift-arm64/tree/master/swift-ci-docker

Sample of Jenkins pipeline I have been using to test docker images and server setup -
https://github.com/futurejones/swift-arm64/blob/master/swift-ci-docker/Jenkinsfiles/Jenkinsfile

@shahmishal The server is setup and should be all ready to go except for your user keys and docker permissions. Will these be the same as before?

[shahmishal]

@futurejones Email sent for additional information

[shahmishal]

Node attached, working on setting up new jobs.

[shahmishal]

@Benkovsky20000 Thanks for reviewing the changes, can you explain the reason for last two comments?

N

docker pull swiftarm/ubuntu_xenial

[futurejones]

@shahmishal @tkremenek - Who is "Benkovsky20000"? - I thought only Mishal could approve PR's for the community CI.

[shahmishal]

Community members can provide feedback, however it does not count toward mergeability.

I have already reported spam comments to GitHub on multiple PRs.

[futurejones]

@tkremenek @shahmishal Is there a reason why this taking so long? It has been over 1 month since I submitted this PR.
If you need help I am more than happy to assist with the Jenkins CI setup.

[shahmishal]

Sorry for taking long time get this completed. This is the first pull request which is going to support Docker/Jenkins Pipeline on ci-external.swift.org, I am working on making sure everything is secure with Docker/Jenkins Pipeline config.

[futurejones]

ok thanks.

[shahmishal]

master branch bots created:

• Amazon Linux 2 - https://ci-external.swift.org/job/oss-swift-RA-amazon-linux-2-aarch64/
• CentOS 8 - https://ci-external.swift.org/job/oss-swift-RA-centos8-aarch64/
• Ubuntu 18.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-18.04-aarch64/
• Ubuntu 20.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-20.04-aarch64/

Node: https://ci-external.swift.org/computer/ubuntu-18.04-aarch64-02/

[shahmishal]

Jenkinsfile:

env.DOCKER_IMAGE, env.OS, env.OS_VERSION are different for each platform.

node('ubuntu-18.04-aarch64-02'){ deleteDir() def DATE = sh(script: "echo `date +%Y-%m-%d`", returnStdout: true).trim() env.SWIFT_BRANCH = 'master' env.SWIFT_SCHEME = 'master' env.SWIFT_VERSION = '5.3' env.RELEASE = 'dev' // change to 'release' for RELEASE versions env.DOCKER_IMAGE = 'swiftarm/amzn_2:latest' env.OS = 'amazonlinux' env.OS_VERSION = '2' env.WORK_DIR = '/home/build-user' stage('Checkout'){ checkout([$class: 'GitSCM', branches: [[name: "*/${SWIFT_BRANCH}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'swift']], submoduleCfg: [], userRemoteConfigs: [[url: 'https://github.com/apple/swift.git']]]) echo "scheme=${SWIFT_SCHEME}: cloning supporting repos" sh "./swift/utils/update-checkout --clone --scheme ${SWIFT_SCHEME}" } stage('Pull Docker Image'){ sh "docker pull ${DOCKER_IMAGE}" } stage('Build Swift'){ try { echo 'Building toolchain' sh "docker rm -f ${JOB_NAME} || true" sh "docker volume rm ${JOB_NAME} || true" sh """ docker run \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ --name ${JOB_NAME} \ -w ${WORK_DIR} \ -v ${WORKSPACE}:/source \ -v ${JOB_NAME}:${WORK_DIR} \ ${DOCKER_IMAGE} \ /bin/bash -lc \ 'cp -r /source/* ${WORK_DIR}; \ ./swift/utils/build-script --jobs=24\ --preset buildbot_linux,no_test \ install_destdir=${WORK_DIR}/swift-install \ installable_package=${WORK_DIR}/output/${JOB_NAME}-${RELEASE}-aarch64-${BUILD_ID}-${DATE}.tar.gz' """ }finally{ sh """ docker rm -f ${env.JOB_NAME} docker volume rm --force ${env.JOB_NAME} || true """ } } } 

[shahmishal]

Sorry it took a while to get it configured, please let me know if you see any issues. Thanks!

[futurejones]

@shahmishal, thanks for getting this done.

There is one small problem with the Jenkinsfile in the installation package path.
installable_package=${WORK_DIR}/output/${JOB_NAME}-${RELEASE}-aarch64-${BUILD_ID}-${DATE}.tar.gz
The output directory needs to be created before the work directory is copied into the docker volume.
or just remove the output directory from the path.
installable_package=${WORK_DIR}/${JOB_NAME}-${RELEASE}-aarch64-${BUILD_ID}-${DATE}.tar.gz

This is causing the builds to fail with the following error the following error -

~/swift-install ~ + tar -c -z -f /home/build-user/output/oss-swift-RA-amazon-linux-2-aarch64-dev-aarch64-7-2020-07-15.tar.gz --owner=0 --group=0 usr/ tar (child): /home/build-user/output/oss-swift-RA-amazon-linux-2-aarch64-dev-aarch64-7-2020-07-15.tar.gz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: /home/build-user/output/oss-swift-RA-amazon-linux-2-aarch64-dev-aarch64-7-2020-07-15.tar.gz: Cannot write: Broken pipe tar: Error is not recoverable: exiting now 

[futurejones]

Also the other Swift - Ubuntu 18.04 Linux aarch64 (master) smoketest job is saying This project is currently disabled.
Is this a naming conflict? Should we change this job name to Swift - Ubuntu 18.04 Linux aarch64 (master/smoketest) or something similar?
https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-18.04-aarch64/

[shahmishal]

@futurejones Updated following jobs by removing output from the path.

• Amazon Linux 2 - https://ci-external.swift.org/job/oss-swift-RA-amazon-linux-2-aarch64/
• CentOS 8 - https://ci-external.swift.org/job/oss-swift-RA-centos8-aarch64/
• Ubuntu 18.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-18.04-aarch64/
• Ubuntu 20.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-20.04-aarch64/

Re-enabled Ubuntu 18.04 job with smoke test in the title.

• https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-18.04-aarch64/