Virtual environments in Python

The venv module in Python creates a light virtual environment that allows users to isolate specific versions of the Python interpreter and software libraries for their projects. The venv module comes with the standard library of Python and does not require a separate installation. Venv is ideal for project-based workflows.

There are a few advantages to working with virtual environments:

• They encapsulate project specific dependencies to prevent any updates to packages to break codes for different projects
• They prevent dependency conflicts
• They allow for reproducibility

Create a virtual environment with venv

To create a virtual environment with venv, enter the following command in your command line:

python -m venv /PATH/TO/NEW/ENVIRONMENT

The virtual environment is created in the /PATH/TO/MY/NEW/ENVIRONMENT directory, so be sure to replace /PATH/TO/MY/NEW/ENVIRONMENT with directory path that you want your environment to be created in. This command also creates any intermediary directories if they do not exist already.

Activate the environment

To activate the new environment, use the source command:

source /PATH/TO/MY/ENVIRONMENT/bin/activate

Be sure to replace PATH/TO/MY/ENVIRONMENT with the actual path to the directory of your virtual environment. The terminal displays the name of your environment in parenthesis, which means your environment is safely activated.

Install packages

Within your virtual environment, you can use common installation tools such as pip to install packages into your virtual environment.

To download any packages that you need for your environment, use the pip command followed by the package(s) you want to install. The following example installs the NumPy and Pandas packages:

pip install numpy pandas

These packages are installed in the venv environment

Deactivate the virtual environment

To deactivate the virtual environment, use the deactivate command:

deactivate

Move your venv virtual environment

The venv environment itself is not portable, but it can be easily recreated in another location from the requirements.txt file. The following steps will show you how to create a new environment called MY_NEW_ENV and download the packages from the requirements.txt file.

1. If you haven’t already, create a requirements.txt file using the following command:

   pip freeze > requirements.txt
   

2. To load the correct python version, use the following command:

   module load python/3.11.0
   

3. To create the venv environment, use the following command:

   python -m venv MY_NEW_ENV
   

4. To install packages from the requirements.txt file, use the following command:

   pip install -r requirements.txt
   

5. To activate the environment, use the following command:

   source MY_NEW_ENV/bin/activate
   

Submit a slurm job with venv

To submit a sbatch job using a venv environment, you can source the environment at the top of the sbatch script.

The following example assumes there is a script called testscript.py that needs to be submitted. This example script prints the versions of the packages we have downloaded in the Move your venv virtual environment example environment called MY_NEW_ENV.

testscript.py

import numpy as np
import pandas as pd
import sys

print(f"Python version = {sys.version}")
print(f"Numpy version = {np.version.version}")
print(f"Pandas version = {pd.__version__}")

To submit this script using a venv environment, create the following script and save it as main.sh.

main.sh

#!/usr/bin/bash
#SBATCH --partition=cpu-preempt      # Partition (queue) name
#SBATCH --ntasks=1                   # Number of CPU cores
#SBATCH --nodes=1                    # Number of nodes
#SBATCH --mem=1gb                    # Job memory request
#SBATCH --time=00:01:00              # Time limit hrs:min:sec
#SBATCH --output=pyenv_test_%j.log   # Standard output and error log

# Load venv env
source /PATH/TO/MY/ENVIRONMENT/bin/activate

# Run script
python testscript.py

Submit the main.sh script to slurm using the command sbatch main.sh. The results are captured in the --output file.

Other virtual environment software options

The following are some additional software options for creating and/or managing virtual environments.

1. Virtualenv is the original and most popular tool to create virtual environments in Python. Venv started as a subset of virtualenv. The biggest advantage is that it works with old python versions.
2. Virtualenvwrapper is a set of extension tools for managing multiple virtualenvs.
3. Pipreqs is a helpful tool for creating requirements.txt files. While pip freeze only logs packages installed with pip, pipreqs will do a more comprehensive job in creating your requirements file and only includes packages that are actively used in your project.