Tutorials

Dask cluster

Start a cluster with distributed

There are multiple ways to create a dask cluster, the following is only an example. Please consult the official documentation.

from dask.distributed import Client

client = Client(n_workers=2, threads_per_worker=2, memory_limit='1GB')

Visualize the cluster: Dashboard

If you are looking for a nice visualization tool, we already have installed this jupyterlab extension into Jupyterhub and is available for all users.

_images/dask-labextension.gif

Sometimes, it is necessary to update the dashboard link for the dask cluster. This can be achieved in either directly in your code or in one of dask configuration files. For example, if you use distributed to start the cluster, you can update the dashboard link in the distributed.yaml in ~/.config/dask directory:

distributed:
   dashboard:
     link: "{JUPYTERHUB_SERVICE_PREFIX}/proxy/{port}/status"

Dask jobqueue

coming soon …