Kernels¶
Here you are going to learn how to use default kernels and how to enable your customized environments
based on e.g conda or virtualenv. Please contact support@dkrz.de
for any issue or request.
System kernels¶
On Mistral, we povide the following Python kernels which are based on Python modules:
- Python 3 / Unstable
it always point to the latest python module
based on this new concept (see details)
R 4.0.3
Julia 1.5.2 (using the module julia/1.5.2-gcc-9.1.0)
ESMValTool (based on the latest esmvaltool module, currently 2.2.0)
From November 15, 2021, the following kernels become deprecated (see here and here):
Python 3 bleeding edge (using the module anaconda3/bleeding_edge)
Python 3 (using the module Python/3.5.2)
Python 2 bleeding_edge (using the module anaconda2/bleeding_edge)
Python 2 (using the module python/2.7.12)
Most of these kernels/modules are based on conda, so you can use conda list
to check all installed packages and their versions.
Note
It is not possible for users to install or update packages. Sometimes you can use the --user
flag to install new packages in your $HOME
directory but there is no guarantee that it will work.
Note
For testing new libraries/packages, we suggest that you use your own conda/env and you turn it into a kernel. If you/we think that a library/package is required by many users or interesting for data processing/visualization workflows, we will add it to one of the above kernels.
Note
Requests for new packages go through our system department and might take a time.
Use your own kernel¶
To get full control of the Python interpreter and packages, we recommend that you create your own environment. PLease, follow these steps:
Note
Some users are experiencing some issue with Miniconda and Python 3.9, we recommend that you load python3/unstable
module to enable conda.
With conda
You need to create a conda environment e.g in your $HOME
directory:
% mkdir $HOME/kernels
% module load python3/unstable
% conda create --prefix $HOME/kernels/your-kernel ipykernel python=3.x
% source activate $HOME/kernels/your-kernel
% python -m ipykernel install --user --name new-kernel --display-name="new kernel"
% conda deactivate
(Re)start the server and the new kernel should be available.
Kernel specifications are in ~/.local/share/jupyter/kernels/
.
More details on kernels can be found here
With virtualenv
If virtualenv is not available, you have to install it first before trying the following steps. The best way to install virtualenv is with pip:
% module load python3/unstable
% python -m pip install --user virtualenv
% python -m virtualenv --system-site-packages /path/to/new-kernel
% source /path/to/new-kernel/bin/activate
% pip install ipykernel
% python -m ipykernel install --user --name new-kernel --display-name="new-kernel"
You can now add/install additional packages that you need in your new environment and then:
(new-kernel) % deactivate
Advanced¶
You can even go further with the configuration of your new kernel by updating the kernel.json. The content looks like this:
{
"argv": [
"/home/user/kernels/new-kernel/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "new-kernel",
"language": "python"
}
It is possible to specify additional environment variables:
{
"argv": [
"/home/user/kernels/new-kernel/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "new-kernel",
"language": "python",
"env": {
"variable": "value",
}
}
Best practices¶
Where to install the new environment?
Depending on the number and size of the python packages installed in your new enviroment, disk usage in your $HOME can easily exceed the limit.
Therefore, it is preferable to create conda/virtualenv environments in /work
(in the corresponding project).
In that case, creating a conda env for example in work:
% conda create --prefix /work/project_id/$USER ...
Helper script for kernel.json
You can create a start-kernel.sh shell script and make it executable (chmod +x start-kernel.sh). Inside the script you can put all configuration you want in your new kernel. An example can be for example loading system modules. The sctructure of the script can be like this:
#!/bin/bash
source /etc/profile
module purge
module load netcdf_c/4.3.2-gcc48
module load python/3.5.2
python -m ipykernel_launcher -f "$1"
And the kernel.json:
{
"argv": [
"start-kernel.sh",
"{connection_file}"
],
"display_name": "new-kernel",
"language": "python"
}
uninstall/remove a kernel
jupyter kernelspec remove kernel
delete the corresponding conda env if you don’t need it anymore