All Posts

Requested MovieWriter (ffmpeg) not available

Do you want to create videos / animations with ffmpeg from your jupyter notebook? you need ffmpeg-python (conda) which requires ffmpeg software on Mistral (module)

conda env with ffmpeg-python and ipykernel

Read more ...

How to containerIze your jupyter kernel?

We have seen in this post how to encapsulate a jupyter notebook (server) in a singularity container . In this tutorial, I am going to describe how you can run a jupyter kernel in a container and make it available in the jupyter*.

Possible use case for this is to install a supported PyTorch version and work with jupyter notebooks (see this).

Read more ...

Create a kernel from your own Julia

If you are using your own Julia installation following (correctly) the instructions described here:

you can use it in Jupyterhub.

Read more ...

Connect Spyder IDE to a remote kernel on Mistral

I am just describing spontaneously what worked for me to connect my local Spyder instance to a remote node on Mistral THAT YOU CAN CONNECT TO VIA SSH FROM YOUR LOCAL MACHINE!!!!

This is just a draft tutorial that will be updated/optimized afterwards.

Read more ...

Python environment locations

Kernels are based on python environments created with conda, virtualenv or other package manager. In some cases, the size of the environment can tremendously grow depending on the installed packages. The default location for python files is the $HOME directory. In this case, it will quickly fill your quota. In order to avoid this, we suggest that you create/store python files in other directories of the filesystem on Mistral.

The following are two alternative locations where you can create your Python environment:

Read more ...

How to quickly create a test kernel

This is a follow up on Kernels. In some cases, the process of publishing new Python modules can take long. In the meantime, you can create a test kernel to use it in Jupyterhub. Creating new conda environments and using them as kernels has been already described here. In this example, we are not going to create a new conda env but only the kernel configuration files.

in this tutorial, I will take the module python3/2021-01. as an example.

Read more ...

Support/Extension for Plotly Python added

The plotly Python library is an interactive, open-source plotting library that supports over 40 unique chart types covering a wide range of statistical, financial, geographic, scientific, and 3-dimensional use-cases.

For a better use of the library in Jupyter Notebook and Lab, extensions are required.

Read more ...

CF Python package added to the software tree

According to this link:

The Python cf package is an Earth Science data analysis library that is built on a complete implementation of the CF data model. The cf package implements the CF data model 1 for its internal data structures and so is able to process any CF-compliant dataset. It is not strict about CF-compliance, however, so that partially conformant datasets may be ingested from existing datasets and written to new datasets. This is so that datasets that are partially conformant may nonetheless be modified in memory.

Read more ...

Update on Jupyterhub @ DKRZ

Upgraded Jupyterlab to v3.0.4

Upgraded the Dask Labextension to v5.0.0 (compatible with lab 3.0)

Read more ...

SLURM update / Memory use

Slurm config on Mistral has been updated to fix an issue related to memory use.

Prior the update, some Slurm jobs continue consuming the available memory (and even swap) of the allocated node and exceed the allocated memory (set in sbatch or srun). If this occurs, it also affect other jobs/users.

error message

Read more ...

Dask jobqueue on Mistral

According to the official Web site, Dask jobqueue can be used to deploy deploy Dask on job queuing systems like PBS, Slurm, MOAB, SGE, LSF, and HTCondor. Since the queuing system on Mistral is Slurm, we are going to show how to start a Dask cluster there. The difference is that the workers can be distributed through multiple nodes from the same partition. Using Dask jobqueue will Dask cluster as a Slurm jobs.

In this case, Jupyterhub will often play an interface role and the Dask can use more than the allocated resources to your jupyterhub session (profiles).

Dask jobqueue

Read more ...

Jupyter notebook/lab extensions

Summary: How to install/enable extensions in Jupyterhub

Extensions bring additional interesting features to Jupyter*. Depending on the workflow in the notebook, users can install/enable extensions when required. Although is easy to add extensions to both Jupyter notebook an lab, the process can be sometimes annoying based on where jupyter is served from.

Extensions configurator

Read more ...

Enable NCL Kernel in Jupyterhub

Summary: How to enable NCL Kernel in Jupyterhub

can’t use NCL (Python) as kernel in Jupyter

Read more ...

Single jupyter notebooks in containers

How can you start single jupyter notebooks in singularity containers?

code is pushed to the original repository

Read more ...

Spawner options now savable

Summary: We added a new feature to the spawner page to save user options.

This feature works also with the preset options form.

Read more ...

New Singularity module deployed

Summary: A new version of Singularity is now available.

Recently, we deployed a new version of Singularity: 3.6.1. The old version is not available anymore due to many bugs reported by some users.

Read more ...

Connection error in your code?

Please check this link for more details on partitions and their limits.

Connection Error

Read more ...

VS Code Remote on Mistral

VS Code Remote development on Mistral.

vs code is your favorite IDE

vs code remote architecture

Read more ...

Jupyterhub log file

Summary: Is Jupyterhub log file name and path annoying for you?

Each Jupyter notebook is running as a SLUM job on MIstral. By default, stdout and stderr of the SLURM batch job that is spawned by Jupyterhub is written to your HOME directory on the HPC system. In order to make it simple to locate the log file:

Read more ...

GLIBC and the container-based workaround

Have you ever tried to install/use a software on Mistral and seen a message like this?

This is for example one of the reasons why PyTorch is not available in our python3 module. Those software packages require a newer version of glibc. Unfortunately, most of Mistral nodes are based on CentOS 6 kernel. To check the version of glibc:

Read more ...

Simple Dask clusters in Jupyterhub

Summary: How to use Dask distributed on Mistral.

There are multiple ways to create a dask cluster, the following is only an example. Please consult the official documentation. The Dask library is installed and can be found in any of the python3 kernels in jupyterhub. Of course, you can use your own python environment.

Dask Labextension

Read more ...

Xarray with Dask Arrays

Starting the Dask Client is optional. It will provide a dashboard which is useful to gain insight on the computation.

The link to the dashboard will become visible when you create the client below. We recommend having it open on one side of your screen while using your notebook on the other side. This can take some effort to arrange your windows, but seeing them both at the same is very useful when learning.

Read more ...