Virtual Python environments are a great way to bundle a collection of libraries for a specific research task or project. We provide a brief guide on how to create virtual Python environments on our JupyterHub, and how to install toolboxes in these virtual environments.

Virtual Python Environments

Introduction and motivation

Our compute server system provides access to a Python environment via JupyterHub. Python is a popular and versatile programming language that has become very popular in data science due to its ease of use, extensive libraries, and flexibility. The Python programming language provides an efficient and streamlined approach to handling complex data structures and gaining insights.

Python has built-in mathematical libraries and functions that make it easy to solve mathematical problems and perform data analysis. In addition, the functionality can be extended by installing a wide range of software toolboxes. The Python system provides the ability to define virtual environments.

Virtual Python environments provide a great opportunity to bundle a collection of libraries for a specific research task or project. In the spirit of persistence of science, virtual Python environments allow to store configurations of toolboxes with versioning and simplify the reproducibility of analysis results.

On our computer server we provide a base configuration of Python in a recent version. In addition, users can create custom configurations of toolboxes using virtual environments. The following is a brief guide on how to create virtual Python environments and how to install toolboxes in these virtual environments.

Step-by-step guide to the installation of a virtual environment

Log in to the KL’s JupyterHub

First, we navigate to the JupyterHub login page and enter our login information here.

Once we have successfully logged in to JupyterHub, the Launcher window will appear. Here we open a terminal. To do this, click on the button marked with a red frame in the next image.

A terminal window opens, as shown in the following figure.

Creating a virtual Python environment

We would like to create a virtual Python environment called ‘my-env’. To do this, we enter the following command in the terminal.

1python3 -m venv my-env

This command creates the new environment ‘my-env’. In addition, a subdirectory ‘my-env’ is created in the user’s home directory. All the toolboxes of the new virtual environment are stored in this directory.

So far, we have only created a virtual environment, but we have not yet assigned a computing engine to it. We want to use Python version 3. We do this by using the following command in the terminal.

1python3 -m ipykernel install --user --name=my-env

If we now open a launcher window (Keyboard shortcut <Ctrl+Shift+L>), two new items are available, red marked in the following image.

New Jupyter notebooks can be created using the ‘my-env’ button in the ‘Notebook’ section. The newly created virtual Python environment ‘my-env’ will be assigned to these notebooks as the computational environment. Similarly, a Python console running in the new compute environment can be opened by pressing the corresponding button in the ‘Console’ section.

Configuring the new virtual Python environment

The newly created virtual Python environment can be customised according to the user’s needs. This enables custom toolbox configurations to be created. As an example, we will install the ‘matplotlib’ toolbox into the newly created virtual Python environment. We will use the ‘pip’ command to do this. This command can be used to install all toolboxes from the Python Package Index (https://pypi.org).

We open the console that uses the newly created virtual environment, see previous figure. First, we need to make sure that the three toolboxes ‘pip’, ‘setuptools’ and ‘wheel’ are available in the current version. To do this, we enter the following command in the Python console that is already open. The command is executed with <Shift+Enter>.

1pip install --upgrade pip setuptools wheel

We then install the matplotlib toolbox in the virtual environment using the following command (execute using <Shift+Enter>).

1pip install matplotlib

The toolboxes that are required to use matplotlib are installed automatically, see the next figure.

All options provided by the ‘pip’ command can be used to maintain the virtual environment, such as install (including version selection), upgrade, delete, etc. For more information on the ‘pip’ command, see https://pip.pypa.io/en/stable/user_guide/.

Removing virtual environments from the Jupyterhub that are no longer required

It is possible to remove unused virtual environments from JupyterHub (in this example, the virtual environment ‘my-env’). To do this, open a console via the JupyterHub launcher (see previous images) and enter the following command. The security question must be answered with ‘y’.

1jupyter kernelspec uninstall 'my-env'

Finally, the subdirectory for the virtual environment created in the home directory should be deleted using the following command in the console.

1rm -rf my-env