Page MenuHomePhabricator

[RFC] on demand resource instantiation
AbandonedPublic

Authored by alangenfeld on May 24 2019, 8:12 PM.

Details

Reviewers
None
Summary

Solids declare which resources they need - so lazily construct resources when we need them instead of all of them at pipeline init.

Test Plan

buildkite

Diff Detail

Repository
R1 dagster
Branch
multicore (branched from master)
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

alangenfeld created this revision.May 24 2019, 8:12 PM
alangenfeld updated this revision to Diff 1017.May 24 2019, 8:30 PM

pipeline_module_fn

does this work?

alangenfeld updated this revision to Diff 1173.May 29 2019, 4:17 PM

backout multicore changes

alangenfeld retitled this revision from run expensive tests with multicore to [RFC] on demand resource instantiation.May 29 2019, 4:18 PM
alangenfeld edited the summary of this revision. (Show Details)
alangenfeld edited the test plan for this revision. (Show Details)

Things to resolve:

  • resource failure can happen at any point in the pipeline
  • custom types have access to resources but do not declare what they need
alangenfeld abandoned this revision.Thu, Oct 3, 10:19 PM