Since PEX v2.1.25 PEX has supported a --venv flag when creating a PEX. The flag causes the created PEX to extract itself into a virtual environment on startup. This has two benefits:
If a PEX is built for many platforms, then the work to determine which wheels to use is done only once when creating the virtualenv.
There is no overhead from unzipping the PEX on startup.
The speed improvements are significant for tools that one might want to run repeatedly such as black or mypy.
Here is a benchmark of running black from a traditional PEX vs one built with the --venv flag.
The PEX built with the --venv flag is almost 3x faster to execute.
Where is the Virtualenv ?
By default the PEX will create a virtualenv in the ~/.pex/venvs directory similar to where PEX keeps its various caches. The full file path is determined by the hash of the PEX as well as the hash of the environment used to select the Python interpreter. Like everything in ~/.pex this can be safely deleted without issue and the next run of the PEX will recreate the folder.
Alternative Location for the Virtualenv
The default location of ~/.pex/venvs is hard to use since the full path of the virtualenv is composed of opaque file hashes and makes it hard to use the created virtualenv with other tools. Alternatively the virtualenv can be created in an arbitrary location using the ‘PEX Tools’ functionality.
When a PEX is created with the --venv flag, the created PEX has some extra PEX related code that can run when the PEX_TOOLS=1 environment variable is set.
The PEX can be created in the ./venv directory by invoking the PEX like so:
What’s in the Virtualenv?
The created virtualenv has everything a regular virtualenv would have plus a few PEX specific extras:
Inside the bin directory we have the activation scripts as well as all of the console entry points from all of the wheels as well as our wheel console scripts.
It’s important to note that setuptools and pip are not installed into this virtual environment which makes them immutable.
We can directly execute a console script in the ./bin directory for an even larger performance boost than executing the PEX built with --venv.
There is also a PEX-INFO file in the root which is copied from the PEX that created the virtualenv.
Finally there is also a __main__.py at the root which makes the entire directory executable by Python. Pointing Python to this directory is equivalent to pointing Python to the PEX file itself except you get the speed benefits of the virtualenv.
Creating a PEX with the --venv flag allows for a virtualenv to be implicitly or explicitly created from the PEX. Creating a virtualenv has less execution overhead than a plain PEX and the resulting virtualenv is immutable.