Skip to content

Dask

Deploy the Dask Services

Stop your currently running containers started with run_dev.sh and then execute run_dev_dask.sh. Don't forget chmod +x run_dev_dask.sh on unix.

What is Dask?

The Dask Tutorial and this article by NVIDIA has decent infographics and explanations on what Dask is. The VERY summarized explanation is it's a library that combines Tornado and Pandas so that an arbitrary number of Python interpreters and Pandas DataFrames can be used as if they were a single interpreter and DataFrame.

The Journey of a Task explanation by the Dask authors provides a nice end-to-end primer on how the framework operates.

It is also worth noting that the Ray Python GPU acceleration framework is rapidly growing in popularity. It can be used as a standalone framework or in conjunction with Dask, Celery, Prefect, Airflow, and other Python data engineering tools.

What is Coiled and Prefect?

Dask fits into a growing segment of the data/tech industry where Free and Open Source Software (FOSS) is provided with fully-managed and extended offerings made available by the primary contributors to make an income.

Two of the more prominent companies aligned with Dask are Coiled.io and Prefect. Coiled is basically a fully-managed Dask cluster while Prefect is an expanded offering more geared towards ETL pipelines.

Dask created hands-on crash course

Dask

Dask

Jupyter Notebook

Transition to the official crash-course running on your computer to get comfortable with the framework.

Jupyter

Your login token will be listed in the terminal next to an avengercon-dask-notebook log entry. You'll need to copy-paste just the token portion of the url

avengercon-dask-notebook       | [... ServerApp] Jupyter Server 2.12.5 is running at:
avengercon-dask-notebook       | [... ServerApp] http://fd52fdf68911:8888/lab?token=b952e22de792f69923d281c04f66393518cd74a0c7fd1acf
                                            EVERYTHING AFTER THE = IS YOUR TOKEN    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

avengercon-dask-notebook       | [I 2024-02-17 19:42:45.576 ServerApp]     http://127.0.0.1:8888/lab?token=b952e22de792f69923d281c04f66393518cd74a0c7fd1acf
avengercon-dask-notebook       | [I 2024-02-17 19:42:45.576 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).