HomePhabricator

RFC: Add support for specifying Dask resources

Authored by Nate Kupp <nate@elementl.com> on Aug 20 2019, 11:31 PM.

Description

RFC: Add support for specifying Dask resources

Summary:
Proposal for programmatic specification of Dask resources. (tests will fail here because the test Dask cluster doesn't provide a "CPU" resource).

Dask resources are abstract and can be anything you want; e.g. {"CPU": 12} is fine, but so is {"foo_bar": 2}; https://distributed.dask.org/en/latest/resources.html - and are specified at worker creation and task instantiation via client.submit()

Worth in the future doing this through the config system, but something like this would at least unblock the folks who are requesting this feature in #general

Test Plan: unit

Reviewers: #ft, schrockn, alangenfeld

Reviewed By: #ft, alangenfeld

Subscribers: schrockn, alangenfeld

Differential Revision: https://dagster.phacility.com/D851

Details

Committed
Nate Kupp <nate@elementl.com>Aug 22 2019, 4:55 PM
Reviewer
Restricted Project
Differential Revision
D851: RFC: Add support for specifying Dask resources
Parents
R1:0585f4650172: Set dagster home in dagit service
Branches
Unknown
Tags
Unknown