Addition January 2019: If you are coming back to this blog after upgrading to macOS Mojave please see this github issue for a solution to the common pyenv ‘zlib not available’ problem.
Before we start, let’s briefly go over the terms used in the title:
- Multiple Python versions: Different installations of Python on the same machine, 2.7 and 3.4 for example.
- Virtual environments: isolated independent environments that can have both a specific version of Python and of any project-specific packages installed within them, without affecting any other projects.
Here we’ll look at three different tools for working with these, and when you might need each one. Let’s explore the use cases for:
If you are using a single version of Python say version 3.3+, and want to manage different virtual environments, then
venv is all you need.
If you want to use multiple versions of Python at 3.3+, with or without virtual environments, then continue to read about
If you also want to work with Python 2, then
pyenv-virtualenv is a tool to consider.
From Python 3.3+ the
venv package is included. It is ideal for creating lightweight virtual environments.
Up until Python 3.6 a script called
pyvenv was also included as a wrapper around
venv, but this has been deprecated. It will be completely removed in Python 3.8. The exact same functionality is available when using
venv, and any existing documentation should be updated. For anyone interested you can read the reasons behind depreciating
venv is used to create a new environment via the terminal command:
$ python3 -m venv directory-name-to-create
$ source name-given/bin/activate
and deactivated with simply:
If you need to remove the environment completely after deactivating it, you can run:
$ rm -r name-given
By default, the environment it creates will be the current version of Python you are using. If you are writing documentation, and want the additional safety that the correct version of Python is being used by your reader you can specify the major and minor version number in the command, like so:
$ python3.6 -m venv example-three-six
If the reader is using a version other than 3.6 then the command will not be successful and will indicate in its error message. However, any patch version (for example 3.6.4) will work.
When the environment is active, any packages can be installed to it via
pip as normal. By default, the newly created environment will not include any packages already installed on the machine. As
pip itself will not necessarily be installed on the machine. It is recommended to first upgrade
pip to the latest version, using
pip install --upgrade pip.
Projects will commonly have a
requirements.txt file specifying its dependencies. This allows the shortcut command
pip install -r requirements.txt command to quickly install all packages to the newly created virtual environment. They will only exist in the virtual environment. It will not be available when it is deactivated but will persist when it is reactivated.
If you do not need to use additional versions of Python itself, then this is all you need to create isolated, project specific, virtual environments.
If you wish to use multiple versions of Python on a single machine, then
pyenv is a commonly used tool to install and switch between versions. This is not to be confused with the previously mentioned depreciated
pyvenv script. It does not come bundled with Python and must be installed separately.
pyenv documentation includes a great description of how it works, so here we will look simply at how to use it.
Firstly we will need to install it. If using Mac OS X, we can do this using Homebrew, else consider other installation options.
$ brew update $ brew install pyenv
Next, add the following towards the bottom of your shell scripts to allow
pyenv to automatically change versions for you:
eval "$(pyenv init -)"
To do, open your in use shell script, via
$ ~/.bashrc or
$ ~/.bash_profile and copy and paste the above line in.
pyenv versions will show which Python versions are currently installed, with a
* next to the one currently in use.
pyenv version shows this directly, and
python --version can be used to verify this.
To install an additional version, say
3.4.0, simply use
pyenv install 3.4.0.
pyenv looks in four places to decide which version of Python to use, in priority order:
PYENV_VERSIONenvironment variable (if specified). You can use the
pyenv shellcommand to set this environment variable in your current shell session.
- The application-specific
.python-versionfile in the current directory (if present). You can modify the current directory's
.python-versionfile with the
- The first
.python-versionfile found (if any) by searching each parent directory, until reaching the root of your filesystem.
- The global version file. You can modify this file using the
pyenv globalcommand. If the global version file is not present, pyenv assumes you want to use the "system" Python. (In other words, whatever version would run if pyenv weren't in your
When setting up a new project that is to use Python 3.6.4 then
pyenv local 3.6.4 would be ran in its root directory. This would both set the version, and create a
.python-version file, so that other contributors’ machines would pick it up.
The full description of
pyenv commands is one to bookmark.
pyenv and venv
When working with Python 3.3+ we now know both how to install and switch between different versions of Python, and how to create new virtual environments.
As an example, let’s say we were setting up a project that was to use Python 3.4.
First we could set our local version using
pyenv local 3.4.0.
If we then ran
python3 -m venv example-project a new virtual environment would be set up under
example-project, using our locally enabled Python 3.4.0.
We activate using
source example-project/bin/activate and can start working.
Next we could optionally document that a collaborator should use
python3.4 -m venv <name>. This means even if a collaborator was not using pyenv the
python3.4 command would error if their Python version was not the same major and minor version (3 and 4), as we intended.
Alternatively, we could choose to simply specify that 3.4.0 was to be used, and instruct
python3 -m venv <name>. If we believe that any version greater than 3.4 is acceptable, then we also may choose to use
python3.4, as if the collaborator was using 3.6 then they would otherwise also receive an error. This is a project specific decision.
pyenv can be used to install both Python 2 and 3 versions. However, as we have seen,
venv is limited to versions of Python greater than 3.3.
pyenv-virtualenv is a tool to create virtual environments integrated with
pyenv, and works for all versions of Python. It is still recommended to use the official Python
venv where possible. But if, for example, you’re creating a virtual environment based on
2.7.13, then this compliments
It also works well with Anaconda and Miniconda
conda environments if you are already using those. A tool called
virtualenv also exists. It’s not covered here, but it’s linked at the end.
pyenv it can next be installed using Homebrew (or alternatives) as so:
$ brew install pyenv-virtualenv
Next in your
.bash_profile (depending on which shell you use) add the following towards the bottom:
eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)"
pyenv to activate and deactivate environments automatically when moving directories.
To create a new virtual environment, use:
$ pyenv virtualenv <version> <name-to-give-it> // for example $ pyenv virtualenv 2.7.10 my-virtual-env-2.7.10
Existing environments can be listed with:
$ pyenv virtualenvs
Activated/ deactivated with:
$ pyenv activate <name> $ pyenv deactivate
At the time of writing, when using
activate the warning
prompt changing will be removed from future release will be displayed. This is expected and refers only to the
(env-name) being displayed in your shell, not the use of the
activate command itself.
Installing requirements works as described in
venv. Unlike in
rm -r command is not needed to remove an environment, an
uninstall command exists.
Between these three tools, we have the ability to collaborate on any project, no matter the version of Python or of the dependencies required. We also know how to document set up instructions for others to use for any project we work on.
We can also see the reasoning behind which set to use, as not all developers will require all three.
Hopefully this was helpful, and is a useful reference in combination with the documentation linked below.
Thanks for reading! ?
Other things I’ve explored:
- Mocking ES and CommonJS modules with jest.mock()
- A beginner’s guide to Amazon’s Elastic Container Service