Python Versions and Dependencies Made Easy with pyenv, venv, pipx and pip-tools

Sebastian Witowski
Python Versions and
Dependencies Made Easy

Sebastian Witowski
Python consultant, freelancer, and trainer
@SebaWitowski switowski.com

Problem with (Python) projects

Stuck in the 90s
Over-engineered

Over-engineered
• Too many shiny new tools from
HackerNews

• Those tools gets abandoned

• You spend too much time
replacing old tools with new ones
Stuck in the 90s

Over-engineered
• Too many shiny new tools from
HackerNews

• Those tools gets abandoned

• You spend too much time
replacing old tools with new ones
Stuck in the 90s
• Everything managed with Bash
scripts

• You have to maintain Bash
scripts or some other scripts
(as a Python developer)

How about a middle ground?

What will we talk about?
• pyenv - installing different Python versions on your computer


• venv - isolating dependencies of your projects



• pipx - installing global packages



• pipx - installing global packages

• pip-tools - dependency pinning

System Python
macOS
Ubuntu 20.04
Debian 10
@SebaWitowski

• Python version preinstalled on your OS is called “system Python”
System Python


• Please, don’t use it
System Python


• Please, don’t use it

• And don’t update it (you might break your OS)!
System Python

How to install a new Python version?

• Use an installer from python.org


• Use your package manager


• Use your package manager

• Compile it from the source files

Or you can use pyenv!
https://github.com/pyenv/pyenv

Or pyenv-win for Windows
https://github.com/pyenv-win/pyenv-win

Installing pyenv

Using pyenv
1. Find Python version to install
$ pyenv install --list
Available versions:
2.1.3
2.2.3
2.3.7
2.4.0
2.4.1
2.4.2
2.4.3
2.4.4
2.4.5
2.4.6
2.5.0
2.5.1
2.5.2
2.5.3
2.5.4
...
...
3.9.0
3.9-dev
3.10-dev
activepython-2.7.14
activepython-3.5.4
activepython-3.6.0
anaconda-1.4.0
anaconda-1.5.0
anaconda-1.5.1
anaconda-1.6.0
anaconda-1.6.1
anaconda-1.7.0
anaconda-1.8.0
anaconda-1.9.0
...
...
miniconda3-4.7.12
pypy-c-jit-latest
pypy-c-nojit-latest
pypy-dev
pypy-stm-2.3
pypy-stm-2.5.1
pypy-1.5-src
pypy-1.5
pypy-1.6
pypy-1.7
pypy-1.8
pypy-1.9
pypy-2.0-src
pypy-2.0
...
...
pypy3.6-7.3.0
pypy3.6-7.3.1-src
pypy3.6-7.3.1
pyston-0.5.1
pyston-0.6.0
pyston-0.6.1
stackless-dev
stackless-2.7-dev
stackless-2.7.2
stackless-2.7.3
stackless-2.7.4
stackless-2.7.5
stackless-2.7.6
...

Using pyenv
2. Install it
$ pyenv install 3.9.0
...
# Wait for a few minutes
...
# Done!

Using pyenv
2. Install it
3. Check if it’s installed correctly
$ pyenv versions
* system
3.9.0

Using pyenv
2. Install it
3. Check if it’s installed correctly
4. Start using this version
$ pyenv global 3.9.0
# or
$ pyenv local 3.9.0
# or
$ pyenv shell 3.9.0

3 levels of pyenv
• pyenv global 3.9.0 - changes the global Python version on your computer

3 levels of pyenv

• pyenv local 3.9.0 - changes Python version for the current folder and all the
sub-folders

3 levels of pyenv

sub-folders
# projectA
$ pyenv local 3.9.0
# projectB
$ pyenv local 3.6.0

3 levels of pyenv

sub-folders

• pyenv shell 2.7.18 - changes Python version for the current shell session

Problem with pip
You can only have one version of a given package installed!

$ pip install Django
...
Django 3.0 installed!
Problem with pip

Problem with pip
$ pip install Django==2.2
...
...
...

$ pip install Django==2.2
...
Django==3.0 uninstalled!
...
Problem with pip

# .../lib/python3.9/site-packages
$ ls -al
...
drwxr-xr-x 22 switowski 704 Oct 30 12:41 django/
drwxr-xr-x 12 switowski 384 Oct 30 12:41 django_redis/
drwxr-xr-x 16 switowski 512 Sep 28 17:58 docutils/
...
Problem with pip

Virtual environment
If pip installs every package in the same folder, why can’t we tell
it to temporarily install them in a diﬀerent folder?

Virtual environment
If pip installs every package in the same folder, why can’t we tell
it to temporarily install them in a diﬀerent folder?
Virtual environment:

• Creates a special folder with Python binaries

• Tells pip to install new packages to that folder

• Tells Python to use packages from that folder

Using virtual environments
1. Create new virtual environment
$ python -m venv .venv

2. Activate it
$ source .venv/bin/activate

2. Activate it
$ source .venv/bin/activate
# Your prompt should change:
(venv) $

2. Activate it
3. Work on your Python project (install packages, run Python)
(venv) $ pip install Django==2.2
...
(venv) $ python manage.py runserver

2. Activate it
3. Work on your Python project (install packages, run Python)
4. Deactivate virtual environment to stop using it
(venv) $ deactivate
$

Typical workflow
# Inside Django 3 project folder
$ python -m venv django3-app
$ source django3-app/bin/activate
$ pip install django==3.0
# Inside Django 2 project folder
$ python -m venv django2-app
$ source django2-app/bin/activate
$ pip install django==2.2

virtualenvwrapper
Windows users: https://pypi.org/project/virtualenvwrapper-win/
Linux and macOS:

virtualenvwrapper
virtualenvwrapper stores all virtual environments in:
~/.virtualenvs/

virtualenvwrapper
virtualenvwrapper stores all virtual environments in:
~/.virtualenvs/
$ mkvirtualenv django2-app - create a virtual environment
$ workon django2-app - activate it
$ lsvirtualenv - list all virtual environments
$ rmvirtualenv django2-app - delete a virtual environment

venv or virtualenvwrapper?

Global Python packages
• Some tools should be installed globally (black, flake8, virtualenvwrapper)


• But if they require diﬀerent versions of the same dependency, one of them
will break


• But if they require diﬀerent versions of the same dependency, one of them
will break

• Installing global tools into separate virtual environments is a hassle (you
have to activate each virtual environment to use that tool)

pipx - global packages in separate environments
https://github.com/pipxproject/pipx

pipx in action
1. Install “black” package
$ pipx install black
installed package black 20.8b1, Python 3.8.5
These apps are now globally available
- black
- black-primer
- blackd
done! ✨ 🌟 ✨

pipx in action
2. Use black
$ black hello_world.py
reformatted hello_world.py
All done! ✨ 🍰 ✨
1 file reformatted.

pipx in action
2. Use black
3. “black” installed in a virtual env will shadow the global one
# Inside a virtual environment
(venv) $ black --version
black, version 19.3b0
# Outside of a virtual environment
$ black --version
black, version 20.8b1

pipx commands
$ pipx list - list all installed packages
$ pipx uninstall <package> - uninstall package
$ pipx upgrade-all - upgrade all packages
$ pipx inject pytest pytest-cov - install pytest-cov inside
pytest virtual env

Dependencies in your project

# requirements.txt
Django
django-redis
pytest

# requirements.txt
Django
django-redis
pytest
$ pip install -r requirements.txt
...
Successfully installed Django-2.2, ...

# requirements.txt
Django
django-redis
pytest
...
...
One month later…

# requirements.txt
Django>=2.2,<3.0
django-redis==4.12.0
pytest==5.*
Dependency pinning:

Always pin dependencies on production servers!

And their sub-dependencies (3rd party packages)!

And don’t forget to update packages regularly!

And don’t forget to update packages regularly!
That’s a lot of work!

Pipenv and Poetry to the rescue!

But do you really need them?

• What if they stop working?


• What if they get discontinued?


• What if they get discontinued?

• Do you really use ALL their features?

Pipenv and Poetry are great

But sometimes a simpler tool is enough

But sometimes a simpler tool is enough
https://github.com/jazzband/pip-tools

pip-compile

pip-compile
# requirements.in
Django>=2.2>,<3.0
pytest
# requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements.in
#
attrs==20.2.0 # via pytest
django==2.2.17 # via -r requirements.in
importlib-metadata==2.0.0 # via pluggy, pytest
iniconfig==1.1.1 # via pytest
packaging==20.4 # via pytest
pluggy==0.13.1 # via pytest
py==1.9.0 # via pytest
pyparsing==2.4.7 # via packaging
pytest==6.1.2 # via -r requirements.in
pytz==2020.4 # via django
six==1.15.0 # via packaging
sqlparse==0.4.1 # via django
toml==0.10.2 # via pytest
zipp==3.4.0 # via importlib-metadata
$ pip-compile requirements.in

pip-compile
# requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements.in
#
attrs==20.2.0 # via pytest
django==2.2.17 # via -r requirements.in
importlib-metadata==2.0.0 # via pluggy, pytest
iniconfig==1.1.1 # via pytest
packaging==20.4 # via pytest
pluggy==0.13.1 # via pytest
py==1.9.0 # via pytest
pyparsing==2.4.7 # via packaging
pytest==6.1.2 # via -r requirements.in
pytz==2020.4 # via django
six==1.15.0 # via packaging
sqlparse==0.4.1 # via django
toml==0.10.2 # via pytest
zipp==3.4.0 # via importlib-metadata

pip-compile
Now, run:

pip-compile
Now, run:
Need to update dependencies?
$ pip-compile requirements.in

pip-compile
Make pip-compile part of your CI

pip-compile
Make pip-compile part of your CI
If something breaks, pin that package in requirements.in

pip-sync

pip-tools
vs.
Pipenv and Poetry

Summary

Summary
• Use pyenv to install/update Python versions

Summary
• Use venv/virtualenvwrapper to isolate dependencies of your projects

Summary
• Use pipx to install global tools

Summary
• Use pipx to install global tools
• Consider pip-tools when you just need to pin dependencies

Where to go next?

Where to go next?
“Modern Python Projects” workshop (part of this conference)

Where to go next?
“Modern Python Projects” online course
https://modernpythonprojects.com/

Python Versions and Dependencies Made Easy with pyenv, venv, pipx and pip-tools

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Python Versions and Dependencies Made Easy with pyenv, venv, pipx and pip-tools

Semelhante a Python Versions and Dependencies Made Easy with pyenv, venv, pipx and pip-tools (20)

Mais de Sebastian Witowski

Mais de Sebastian Witowski (7)

Último

Último (20)

Python Versions and Dependencies Made Easy with pyenv, venv, pipx and pip-tools