Do Ins/Outs need to have names if we're going with the dict approach?

Overall I am on board with this. It still needs docs etc. and I left a comment on one of the details of the API.

nice! I was hoping someone would add this

Mind adding a test for a class that defines an init fn?

@schrockn I’m very on board with the dict format now that I know that 3.6+ dicts are ordered.

up

up

up

Does this change imply that daemon output streams should get handled by the same InstanceLogManager as the step process output streams? Or will they be separately configurable?

up

up

requesting changes for q management

I like SensorEvaluationContext and ScheduleEvaluationContext too.

up

up

up

up

up

I had a conversation with @schrockn and I am receptive to his argument that it's more elegant for jobs to have names than suffixes. Here's a re-spin that does that.

up

lgtm!

up

where do we do partition set look ups? Is there a way to store a different type of targeting on the object that refers to it, like JobScoped instead of RepoScoped so you go get the Job and fetch it of there instead of getting it out of the repo?

awesome

Will this config be used when invoking the job from a schedule or sensor? I had been imagining that it would.

We are not currently omitting it in the header of the Pipeline (soon Job?) page, nor in the URL. But I can omit it here if desired.

After this goes in, what will the procedure be for making changes to the pipelines? Will there be some intermediate period during which we need to replicate changes across both the public repo and the demo repo? Not the end of the world, but ultimately will be pretty painful. Do we have a plan for making the demo depend on the public repo version of the pipeline?

Not blocking for this diff, but we've discussed omitting the mode name when the mode is "default", so I think we'd want to do that here as well if we did that elsewhere.

Yeah my partitions diff definitely has the same issue.

if we did this we would want it behind an opt-in gating of some kind right?

I think the main thing worth discussing here is that this shifts ModeDefinitions from objects that can live inside multiple pipelines to objects that are specific to a single pipeline. In legacy non-crag world, do we think its legitimate for a user to use the config_mapping arg on ModeDefinition, or is this just a hack to get crag working that doesn't require adding JobDefinition, etc.? I'd be more comfortable with the latter than the former.

looks like we don't hard error on io_manager_key being set on composite, should we?

up

Do we want to make the context arg itself optional?

I'm going to defer to @prha

If you look in test_io_manager_composites.py, only the IOManagers on the leaf solids take effect. So my expectation was that for root input managers, we would ban them on composite solids for consistency. Are there reasons that doesn't make sense?

LGTM!

up

up

up

up

Amazing