HomePhabricator

Multiprocessing and Dask execution through Dagit

Authored by Max Gasner <max.gasner@gmail.com> on Aug 5 2019, 9:27 PM.

Description

Multiprocessing and Dask execution through Dagit

Summary: Resolves https://github.com/dagster-io/dagster/issues/1407 and https://github.com/dagster-io/dagster/issues/1554

Test Plan:
Unit, manual:

Execute the sleepy_pipeline from dagit with config as follows:

storage:
  filesystem:
execution:
  multiprocess:
    config:
      max_concurrent: 4
solids:
  giver:
    config:
      - 1
      - 5
      - 10
      - 5

Execute the hammer_pipeline from Dagit with config as follows:

storage:
  filesystem:

execution:
  dask:

(logs do not stream, but progress is visible on command line)

Reviewers: alangenfeld, natekupp, schrockn, #ft

Reviewed By: alangenfeld, schrockn, #ft

Subscribers: sashank

Differential Revision: https://dagster.phacility.com/D771

Details

Committed
Max Gasner <max.gasner@gmail.com>Aug 8 2019, 9:29 PM
Reviewer
Restricted Project
Differential Revision
D771: Multiprocessing and Dask execution through Dagit
Parents
R1:54586608bcb2: change `file_handle_to_s3` solid to use S3.Client.upload_fileobj
Branches
Unknown
Tags
Unknown