This is where the step skipping check happens.

I believe we would like to check output event instead of the presence of the output object. In order to do the check, we need to know if the upstream step is skipped, i.e. if it yields any output event. so it leads us to either

1. ask instance to find out about it based on events
2. figure out some way to pass that info through context (e.g. context.step) to avoid db call. it'd be tricky because ExecutionStep is locked at run time. Or maybe we can somehow inspect the event stream during core_dagster_event_sequence_for_step, which could be risky too because it's gonna be on the very critical path

as far as I can tell, asking the instance is probably the best we can do here. ideally, perhaps we'd find some way to pass it in when we launch the step, but I think that would require some fairly big architectural changes.

updated with the instance call approach (see instance.step_has_output) and tracked the ideal way in an issue https://github.com/dagster-io/dagster/issues/3511

not happy with this fix :( but it seems to be the best we could do atm.

@sandyryza
a minor perf improvement might be to have an in-memory instance dict to keep track of skipped steps per run_id to avoid db interaction. but i dont think it's worth it too much since we already limit the #calls by checking fan-in and optional output.

this is the only callsite

the check happens in self._sources_to_skip now and we now make a single instance call once per fan-in input.

was doing _sources_to_skip in https://dagster.phacility.com/D5855?id=29104 which pre-checked if all sources are required so we could avoid unnecessary instance call.
but it wouldn't work for cases like https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/dagster/dagster_tests/core_tests/test_pipeline_execution.py#L1046:9 where the fan-in step's transitive dependencies are optional but its direct sources are required.

so we either do a look all the way up for each entry for the fan-in step to find out if there's any optional step output in the graph, or we simply do instance call for every fan-in step input. here i chose to the latter

given [1] and [2] as the only other implementations, what do you think about consolidating all of this in to [3] and removing can_load_input_object from the base class

[1]

[1]

[2] i expect this is never invoked

[3]

i would prefer if we not add this function to the instance, and just invoke all_logs from the callsite