pybuild-install

Install command is meant to provide a stopgap solution and is best suited to be handled by other tools. However, fitting those tools easily and natively into the workflows of tools/frameworks for packaging AWS Lambda functions is challenging for users:

Compare & Contrast with NodeJS

• In NodeJS with the esbuild tool - due to the ability to tree-shake any excess code, users can install dependencies once in a central location. esbuild does not concern itself with installing dependencies.
• In NodeJS, it seems less common to have to install dependencies in a "Lambda-like" environment. This greatly simplifies the process of package management to standard tools like NPM, Yarn, etc.
• This leaves the typical CI/CD pipeline for NodeJS artifacts that are bundled to look like:
  • Checkout source code
  • Install dependencies using standard tooling (NPM, Yarn, etc.) within the CI environment
  • Build/bundle artifact using esbuild (this step can be spread out and parallelized for multiple artifacts)
  • Deploy artifacts
• If we contrast this with a Python project
  • Checkout source code

TODOs

• ✅ For pipenv and poetry, how to export only production dependencies?
  • For pipenv, this seems to be the default https://github.com/pypa/pipenv/issues/245
  • For poetry, this also seems to be the default when inspecting poetry export -h
• install This is just a simple wrapper that shells out commands to the cli to ergonomically handle where deps are installed for bundling

  • ✅ --package-manager, -p: one of pip, poetry, or pipfile

    • pip will use dependencies listed in requirements.txt

      pip install -r requirements.txt --target ./.pybuild/site-packages
      

    • pipenv

      • https://github.com/pypa/pipenv/issues/746

      pipenv run pip install -r <(pipenv lock -r) --target ./.pybuild/site-packages
      

    • poetry

      • https://github.com/python-poetry/poetry/issues/1937#issuecomment-612398471

      poetry export -f requirements.txt --without-hashes > requirements.txt & \
      poetry run pip install -r requirements.txt -t ./.pybuild/site-packages
      

  • ✅ --extra-args, -e: extra arguments to pass to package manager install command

  • ✅ --target, -t: Install packages into

    . By default this will not replace existing files/folders in . Use --extra-args='--upgrade' to replace existing packages in with new versions.
    • By default we will use .pybuild/site-packages since this is where pybuild bundle ... will look
    • Clearly document that if users change --target tha they will also want to change --packages to match when bundling

  • ✅ --upgrade, -u: Upgrade all specified packages to the newest available version

  • --requirements-file, r: path to requirements file (requirements.txt, Pipfile, pyproject.toml, etc.)

• ✅ --log-level, -l: log level
• ✅ --version, -v: version

Enhancements

• Add ability for users to --install-in-docker to ensure libraries are compatible for various runtime environments like AWS Lambda
• Allow users to select which container image to use for --install-in-docker using --container-image
• Add flag --python-version to aid in selecting default container image and archive path, etc.

Package Install Location Challenges

Pip

• https://www.python.org/dev/peps/pep-0582/
  • https://discuss.python.org/t/pep-582-python-local-packages-directory/963

Pipenv

• https://github.com/pypa/pipenv/issues/746

Poetry

• https://github.com/python-poetry/poetry/issues/1937