feat: Support installing dependencies with poetry #186
Conversation
| build_in_docker = true | ||
| runtime = "python3.8" | ||
| source_path = "${path.module}/../fixtures/python3.8-app1" | ||
| artifacts_dir = "${path.root}/builds/package_dir/" |
There was a problem hiding this comment.
Specifying different artifacts_dir for each module makes it easier to check lambda packages content.
| "!_virtualenv.pth", | ||
| "!_virtualenv.py", | ||
| "!distutils-precedence.pth", | ||
| "!easy_install.py", | ||
| "!(pip|pip-.*)(\\.virtualenv|/.*)", | ||
| "!(pkg_resources|pkg_resources-.*)(\\.virtualenv|/.*)", | ||
| "!(setuptools|setuptools-.*)(\\.virtualenv|/.*)", | ||
| "!(wheel|wheel-.*)(\\.virtualenv|/.*)", |
There was a problem hiding this comment.
This is required as poetry does not support a --target parameter like pip so everything is put in the same directory of the created virtualenv.
| os.path.join(path, 'requirements.txt')) | ||
| step('zip', path, None) | ||
| if runtime.startswith("python"): | ||
| pip_requirements_step(os.path.join(path, "requirements.txt")) |
There was a problem hiding this comment.
pip_requirements_step() already appends the filename if path is a directory:
| pip_requirements_step(os.path.join(path, "requirements.txt")) | |
| pip_requirements_step(path) |
| shlex_join([ poetry_exec, "config", "--no-interaction", "virtualenvs.create", "true" ]), | ||
| shlex_join([ poetry_exec, "config", "--no-interaction", "virtualenvs.in-project", "true" ]), | ||
| shlex_join([ poetry_exec, "install", "--no-interaction" ]), |
There was a problem hiding this comment.
Until poetry install supports a --target parameter, an alternative may be to let poetry export the dependencies in requirements.txt format then install dependencies with pip:
| shlex_join([ poetry_exec, "config", "--no-interaction", "virtualenvs.create", "true" ]), | |
| shlex_join([ poetry_exec, "config", "--no-interaction", "virtualenvs.in-project", "true" ]), | |
| shlex_join([ poetry_exec, "install", "--no-interaction" ]), | |
| shlex_join([ poetry_exec, "export", "-f", "requirements.txt", "--output", "requirements.txt" ]), | |
| shlex_join([ python_exec, '-m', 'pip', 'install', '--no-compile', '--prefix=', '--target=.','--requirement=requirements.txt'), | |
There was a problem hiding this comment.
@pdecat please add --no-dev to the install command. I believe it's more correct behaviour
There was a problem hiding this comment.
@Tonkonozhenko good point, done!
| I've seen this Pull Request not proceeding since 26th of July @Tonkonozhenko |
| Well, I've not abandoned it yet, it's just blocked on current poetry limitations. Feel free to experiment if you have better ideas to do it properly.
|
| @pdecat One good approach might be generate requirements.txt from poetry and use it. Then requirements.txt can be used by pip_requirements File poetry.toml should also be uploaded inside building container. poetry config --no-interaction virtualenvs.create true poetry config --no-interaction virtualenvs.in-project true poetry export --with-credentials --format requirements.txt --output requirements.txt |
| Tried that, but in some situations, the generated I remember facing issues like those mentioned in https://github.com/python-poetry/poetry/issues/3160 Maybe time to re-evaluate that option. |
pdecat force-pushed the support_poetry branch from 0d174a6 to 93f92f1 Compare | Rebased on current |
| That's really interesting @pdecat. In which case requirements.txt generates issue? I've personally never faced a problem like that. |
| Tried to reproduce the issue with a few projects of mine but I did not manage to reproduce it with poetry 1.1.11. |
| This PR has been automatically marked as stale because it has been open 30 days |
pdecat force-pushed the support_poetry branch from 93f92f1 to 13614be Compare pdecat changed the title pdecat force-pushed the support_poetry branch from 13614be to 3a63ccb Compare 
pdecat force-pushed the support_poetry branch from 3a63ccb to 5ce931c Compare | This PR was automatically closed because of stale in 10 days |
| I'll re-open a new PR once all blocking points are resolved. |
| Another issue I faced may have now been resolved: python-poetry/poetry#3472 (comment) |
| Re-opened as #311 using pip to install dependencies. |
| I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
3 participants
Description
This PR adds support for installing dependencies with poetry for projects with
pyproject.tomland optionallypoetry.lockfiles.Marked as draft as poetry does not currently provide a proper way to isolate installed dependencies from virtualenv (pip's
--targetoption) so this implementation is kind of a hack that requires manually excluding undesired files from the lambda package.Submitted the PR early anyway to get feedback.
Motivation and Context
poetry's is more and more used and has a robust dependency resolver.
Breaking Changes
New poetry install step is not enabled by default and requires explicitly setting
poetry_install = Truein the source block.How Has This Been Tested?
examples/*projects