Page MenuHomeElementl

[crag] schedule & sensor target
ClosedPublic

Authored by alangenfeld on May 18 2021, 2:36 PM.

Details

Summary

A set-up change for direct targeting, orients the current set-up around "repo relative" targets of string pointers

Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Interesting. Are there different combinations of things that are potentially supported here / could you partially specify? Or is the idea that in a post-crag world, all of these things would be coupled into a singular entity?

Are there different combinations of things that are potentially supported here / could you partially specify? Or is the idea that in a post-crag world, all of these things would be coupled into a singular entity?

I think most likely the singular entity - the diffs above will allow

@schedule(
  target=my_graph.with_resources(...),
  ...

Yup, I like using target as a singular noun here. Way easier to think about mode/solid selection.

In the case we end up supporting one-to-many sensors to targets, what would the run request look like? Would we consider exposing RepoRelativeTarget and force users to construct them? Or should the targets themselves have names?

In the case we end up supporting one-to-many sensors to targets, what would the run request look like?

my current idea

@sensor(
  targets={'foo': my_graph.with_resources(...), 'bar': other_graph.with_resources(...)}
 ...
) 
def ok():
  yield RunRequest(
    target_key='foo',
     ...
  )

going with dict/key approach since you may have the same graph with different resources which makes targeting by name likely awkward / tricky

Would we consider exposing RepoRelativeTarget and force users to construct them?

maybe - you would likely only use it on conjunction with lazy-loaded-pipelines (targets) in your repo definition which is an advanced perf optimization kinda thing

Or should the targets themselves have names?

Im open to the idea of accepting target="graph_name", target="graph_name[qualifier_name]", or target=("graph_name", "qualifier_name") instead of having to construct a dingus but also respect the arguments against super flexible input typing

Oh nice, yeah target_key feels pretty good there.

This revision is now accepted and ready to land.May 28 2021, 5:39 PM
This revision was automatically updated to reflect the committed changes.