[CI] Add some more steps to test on Macos ARM #2030

mrodm · 2024-08-21T11:49:02Z

Relates #539

This PR adds a new step in the CI pipeline to test to build packages using elastic-package build command in MacOS ARM VMs. Doing that allows us to check that the at least can be executed this command in that platform and architecture.

For the time being, it is not available docker/docker-compose commands in those MacOS ARM VMs.

mrodm · 2024-08-21T16:53:53Z

/test

mrodm · 2024-08-21T17:31:43Z

.buildkite/pipeline.trigger.integration.tests.sh

+# TODO: Missing docker & docker-compose in MACOS ARM agent image, skip installation of packages in the meantime.
+# If docker and docker-compose are available for this platform/architecture, it could be added a step to test the stack commands (or even replace this one).
+echo " - label: \":macos: :go: Integration test: build-zip\""
+echo " command: ./.buildkite/scripts/integration_tests.sh -t test-build-zip-skip-install"


Could it be interesting to add this step in the CI pipeline ? WDYT ?

Sounds good, yes.

jsoriano

LGTM, but added suggestion about having a test script that just builds the zip.

jsoriano · 2024-08-22T14:10:16Z

.buildkite/pipeline.trigger.integration.tests.sh

+# TODO: Missing docker & docker-compose in MACOS ARM agent image, skip installation of packages in the meantime.
+# If docker and docker-compose are available for this platform/architecture, it could be added a step to test the stack commands (or even replace this one).
+echo " - label: \":macos: :go: Integration test: build-zip\""
+echo " command: ./.buildkite/scripts/integration_tests.sh -t test-build-zip-skip-install"


Sounds good, yes.

jsoriano · 2024-08-22T14:13:25Z

scripts/test-build-zip.sh


+if [[ "${SKIP_INSTALL}" == 1 ]]; then
+ exit 0
+fi


Nit. Instead of having this if and the ones in with_docker and with_docker_compose, maybe we can add a new script that just builds the zip and doesn't call to with_docker and with_docker_compose. Even if we have some duplication between the scripts, it would be more explicit. WDYT?

Sure, I'll do that. It is going to be better to have that script with less "ifs"

jsoriano · 2024-08-22T17:16:33Z

.buildkite/scripts/install_deps.sh

+ if [[ "${platform_type_lowercase}" == "darwin" ]]; then
+ echo "Skip. Not supported in Darwin."
+ return
+ fi


This is not needed now, right?

It is still needed. These function (with_docker and with_docker_compose) are called from the script .buildkite/scripts/integrations_tests.sh. This script is used for all the steps that could run with Linux containers or Linux VMs, so it is needed to keep these safeguards/checks.

elastic-package/.buildkite/scripts/integration_tests.sh

Lines 83 to 88 in 10509f6

echo "--- install docker"

with_docker

echo "--- install docker-compose plugin"

with_docker_compose_plugin

fi

Maybe I am missing something, but do we want these scripts to continue if docker installation cannot be performed?

This script is used for all the steps that could run with Linux containers or Linux VMs, so it is needed to keep these safeguards/checks.

Is it run in darwin?

This script is used for all the steps that could run with Linux containers or Linux VMs, so it is needed to keep these safeguards/checks.

Is it run in darwin?

Ok, I see it is run with -t test-just-build-zip, could we avoid calling with_docker and with_docker_compose_plugin when the target is test-just-build-zip? The same way as we are avoiding it when running for serverless.

Exactly, for now just Macos ARM is used for that specific make target.
It could be added as you mentioned that condition to not call those functions if that target is the one to be executed.

I'll give it a try 👍

Applied the suggestion here 2e7220d

jsoriano · 2024-08-22T17:16:41Z

.buildkite/scripts/install_deps.sh

+ if ! command -v docker &> /dev/null ; then
+ echo "Skip. Docker is not installed by default"
+ return
+ fi


As mentioned here, I would keep this condition too.
In this case, if there is no docker there is no need to install docker-compose.

mrodm · 2024-08-26T15:50:25Z

/test

mrodm · 2024-08-27T08:50:13Z

/test

elasticmachine · 2024-08-27T09:24:56Z

💚 Build Succeeded

Buildkite Build
Commit: 2e7220d

History

💚 Build #3802 succeeded 2e7220d
💛 Build #3801 was flaky 2e7220d
💚 Build #3782 succeeded 27b79be
💚 Build #3776 succeeded 735c1aa
💔 Build #3775 failed 735c1aa
💚 Build #3772 succeeded 27e73ff

cc @mrodm

jsoriano · 2024-09-02T16:48:57Z

Makefile

 test-build-zip:
 ./scripts/test-build-zip.sh


What does this apart of just building the zip? 😅

This script builds the packages and then installs them (without the zip file). It would be briefly like:

elastic-package stack up -v -d # for each package elastic-package build -v --zip --sign elastic-package install -v elastic-package stack down -v

Then, there is another script that builds the packages and installs them using the zip file (scripts/test-install-zip.sh):

elastic-package stack up -v -d # for each package elastic-package build -v --zip --sign elastic-package install -v --zip <zip_file_path> # check via curl that it is installed elastic-package stack down -v

And there is another script file, that just builds the zip file. Used specially for testing in Macos ARM:

elastic-package stack up -v -d # for each package elastic-package build -v --zip --sign elastic-package stack down -v

Probably, they could be renamed so it is clear what they do. What about rename them as this? @jsoriano

scripts/test-build.zip.sh: scripts/test-build-install.sh

Package build and installed from the working copy

scripts/test-install.zip.sh: scripts/test-build-install-zip-file.sh

Package built and installed using the zip file

scripts/test-just-build-zip.sh: scripts/test-build.zip.sh

Package built

If so, I could create a follow-up PR to rename those files as well as Makefile and buildkite scripts accordingly.

Yeah, these scripts are a bit confusing. Let's rename them in a follow up.

Created PR to rename test scripts: #2072

mrodm self-assigned this Aug 21, 2024

mrodm force-pushed the add-some-more-steps-arm branch from 933fe0c to 3694852 Compare August 21, 2024 14:16

mrodm mentioned this pull request Aug 21, 2024

[META] Support ARM in developer pipeline #539

Open

mrodm added 5 commits August 21, 2024 18:17

Test build zip step in macos arm

f9fe8ab

Add safeguards

2307d81

Skip actions in cleanup if not installing packages

0d7f5d9

Restore integration tests

6dd4b2f

Add TODO message

735c1aa

mrodm force-pushed the add-some-more-steps-arm branch from a4cc1fb to 735c1aa Compare August 21, 2024 16:18

mrodm marked this pull request as ready for review August 21, 2024 17:31

mrodm requested a review from a team as a code owner August 21, 2024 17:31

mrodm commented Aug 21, 2024

View reviewed changes

jsoriano approved these changes Aug 22, 2024

View reviewed changes

Create new script file

27b79be

jsoriano reviewed Aug 22, 2024

View reviewed changes

Skip install docker and docker-compose

2e7220d

mrodm changed the title ~~[CI] Add some more steps arm~~ [CI] Add some more steps to test on Macos ARM Aug 27, 2024

mrodm requested a review from jsoriano August 27, 2024 10:05

jsoriano approved these changes Sep 2, 2024

View reviewed changes

mrodm merged commit 8ca8a80 into elastic:main Sep 3, 2024

mrodm deleted the add-some-more-steps-arm branch September 3, 2024 10:08

mrodm mentioned this pull request Sep 3, 2024

[CI] Rename test scripts #2072

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[CI] Add some more steps to test on Macos ARM #2030

[CI] Add some more steps to test on Macos ARM #2030

Uh oh!

mrodm commented Aug 21, 2024 •

edited

Loading

mrodm commented Aug 21, 2024

mrodm Aug 21, 2024

jsoriano Aug 22, 2024

jsoriano left a comment

jsoriano Aug 22, 2024

jsoriano Aug 22, 2024

mrodm Aug 22, 2024

jsoriano Aug 22, 2024

mrodm Aug 26, 2024

jsoriano Aug 26, 2024

jsoriano Aug 26, 2024

mrodm Aug 26, 2024

mrodm Aug 27, 2024

jsoriano Aug 22, 2024

mrodm Aug 26, 2024 •

edited

Loading

mrodm commented Aug 26, 2024

mrodm commented Aug 27, 2024

elasticmachine commented Aug 27, 2024

jsoriano Sep 2, 2024

mrodm Sep 3, 2024

jsoriano Sep 3, 2024

mrodm Sep 3, 2024

Labels

3 participants

	echo "--- install docker"
	with_docker

	echo "--- install docker-compose plugin"
	with_docker_compose_plugin
	fi

Uh oh!

[CI] Add some more steps to test on Macos ARM #2030

[CI] Add some more steps to test on Macos ARM #2030

Uh oh!

Conversation

mrodm commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mrodm commented Aug 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm Aug 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm commented Aug 26, 2024

mrodm commented Aug 27, 2024

elasticmachine commented Aug 27, 2024

💚 Build Succeeded

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Labels

3 participants

mrodm commented Aug 21, 2024 •

edited

Loading

mrodm Aug 26, 2024 •

edited

Loading