Migrate CI to Jenkins on 7.x branch #1114
Conversation
| print("No elasticsearch repo found...") | ||
| # set YAML DIR to empty to skip yaml tests | ||
| environ["TEST_ES_YAML_DIR"] = "" | ||
| return |
There was a problem hiding this comment.
this option was useful to avoid running the rest tests as they take a long time, this will mean that a typo in a setting will cause the entire ES repo to be cloned which is a very expensive process
There was a problem hiding this comment.
That's true, I'll add back a way to disable running API tests. Will also apply that flag onto the oss Jenkins CI config.
There was a problem hiding this comment.
the rest tests that are running now are for the oss release
| # | ||
| # Export the ELASTICSEARCH_VERSION variable, eg. 'elasticsearch:8.0.0-SNAPSHOT'. | ||
| | ||
| # Version 1.0.2 |
There was a problem hiding this comment.
this file should not live in this repo if it is shared by multiple clients, instead it should be in the shared repository somewhere to avoid duplication. It is also unnecessarily complex for the needs of the tests. To run the server for x-pack tests all you need is:
docker run \ --publish 9200:9200 \ --env "xpack.security.authc.token.enabled=true" \ --env "xpack.security.authc.api_key.enabled=true" \ --env "xpack.security.enabled=true" \ --env "ELASTIC_PASSWORD=changeme" \ --env "xpack.license.self_generated.type=trial" \ --env "discovery.type=single-node" \ --env "path.repo=/tmp" \ --env "repositories.url.allowed_urls=http://*" \ --env "node.attr.testattr=test" \ --env "node.name=test" \ docker.elastic.co/elasticsearch/elasticsearch:$VERSION (this does not include SSL which can be added).
For non x-pack tests it is
docker run \ --publish 9200:9200 \ --env "discovery.type=single-node" \ --env path.repo=/tmp \ --env "repositories.url.allowed_urls=http://*" \ --env node.attr.testattr=test \ --env node.name=test \ docker.elastic.co/elasticsearch/elasticsearch:$VERSION not sure why this entire infrastructure is needed and what it adds for the complexity.
There was a problem hiding this comment.
For now this does need to live in this repository as well. The added complexity is in part to support running in different modes. e.g all the clients can run the yaml tests using:
$ ELASTICSEARCH_VERSION=7.x-SNAPSHOT TEST_SUITE=xpack ./.ci/run-tests Or choose to run run-elasticsearch separately and (using DETACH=true or not) during development.
After we get everyone in this harness we'll explore our options to move parts of this into a shared space.
.ci/Dockerfile Outdated
| | ||
| WORKDIR /code/elasticsearch-py | ||
| | ||
| COPY . . |
There was a problem hiding this comment.
You can take caching layers to your advantage by first copying the dependency management files, then call pip install and then do COPY . .
This prevents having to pull the dependencies every time you make a code change.
See e.g:
There was a problem hiding this comment.
A https://github.com/elastic/eland/blob/master/.dockerignore file could further help speed up the spin up of the container.
There was a problem hiding this comment.
Problem is installing . requires many files currently, could split it out into a dev-requirements.txt though for sure. I'll add a .dockerignore file. :)
| Now that the jobs are showing up in Jenkins, going to cycle this PR to see if we can get a build. |
| Now that the jobs are showing up in Jenkins, going to cycle this PR to see if we can get a build. |
4 participants
Based mostly on
elasticsearch-ruby's configuration but adapted for Python.Trying on the 7.x branch first as the test suite doesn't pass against the 8.0 Docker image on
master. Will migratemasterto Jenkins once I 8.0-ify that branch.Also using this to test if Jenkins will pick up this PR, is there any way that can be tested before we merge this?