Here's a comparison of what it would look like to implement the "launch a run for each file in an S3 directory". I think the sensors-5 version is considerably cleaner.

sensors-2:

def run_per_s3_file_config_fn(context):
    all_s3_files = get_all_s3_files_in_directory("some_dir")
    for file in all_s3_files:
        if not run_exists(context, file):
            return {"the_file": file}


@sensor(run_config_fn=run_per_s3_file_config_fn)
def run_per_s3_file(context):
    all_s3_files = get_all_s3_files_in_directory("some_dir")
    for file in all_s3_files:
        if not run_exists(context, file):
            return True

sensors-5:

@sensor
def run_per_s3_file(context):
    all_s3_files = get_all_s3_files_in_directory("some_dir")
    for file in all_s3_files:
        if not run_exists(context, file):
            yield {"the_file": file}

And here's a comparison of what it would look like to implement the "launch a run if there's something at this S3 path". I don't think the sensors-5 version is significantly messier.

sensors-2:

@sensor
def run_if_s3_file():
    return s3_file_exists("some_path")

sensors-5:

@sensor
def run_if_s3_file():
    if s3_file_exists("some_path"):
        yield {}

Another advantage of sensors-5 over sensors-2: If someone wants to have a sensor that can kick off a bunch of runs "at once", sensors-2 would require them to have a very fast tick. Maybe kicking off a ton of runs at once isn't something we want to support? Though I still think there are backfill approaches that could benefit from sensors kicking off a bunch of runs.

Preaching to the choir, @sandyryza

The only downside of this approach is that I prefer having distinct, named params, instead of some dictionary with named keys. This would require another top-level object for folks to used (e.g. JobParams)

sensors-5 probably makes more sense for implementing schedules on top of this as well, we're definitely going to want to potentially kick off multiple runs per 'tick' for those.

Sorry for entering the fray late if there has already been back and forth on the "JobConfig" naming. My two cents:

I think "Config" means something pretty specific in dagster already - i.e. something like "an object that can satisfy a config schema". If we can avoid diluting it, I think that's preferable. Was there pushback on "Params"? I liked that.

Are the objects that are yielded by sensors 1:1 with runs? If so, could we call it RunParams?

Yeah, I'm not too attached to JobConfig, just exploring the naming space. I like having the word Job in there, to cement the concept of these things that job definitions return, that parameterize launched runs. I guess I would prefer JobParams over RunParams.

My impression of the common usage of naming something XConfig or XParams is that X is the thing being configured or parameterized. E.g. "run config" is the configuration for a particular run.

If I'm understanding the meaning of job correctly, it's not the Job itself that is being configured. The sensor is the job, and the sensor itself isn't being configured or parameterized by the produced object.

I'm really confused by JobConfig. From my perspective the current JobConfig in the diff represents the parameters to a run. It represents "I have determined that in this tick there should be a run launcher, and here are the parameters". JobConfig sounds like an argument that would be passed into the *job* definition.

I think it might be helpful to write proposed english language descriptions of runs and jobs to make it clear what we are building so that we are all 100% in agreement about the proposed ontology

Your impression is correct. The current JobConfig is really a 1-to-1 mapping of the dynamically-resolved parameters required to launch a run (run_config, tags). It's a bit annoying to have both of these. Each sensor can resolve to a list of these parameters.

The early iterations of this diff (see https://dagster.phacility.com/D5026?id=25165) just had the sensor body return a list of dictionaries that had run_config and tags as keys, but it feels awkward and error prone to have the body return a dict.

I think the top-level options to explore are:

1. Have sensor bodies return dictionaries and have very explicit error messages if they pass the wrong keys. This keeps tags/run_config separately defined, and both dynamically resolved.
2. Rename JobConfig to something like RunParams, keeping tags/run_config separate, but introducing a noun that contains both. We can redefine the schedule definitions and the partition set definitions to use run_param_fn instead of all having individual run_config_fn, tag_fn, should_execute_fn. Should note that aside from mode, this is essentially a preset.
3. Try to keep only a single concept of run_config and have tags be defined within the run_config. This might be conceptually cleaner, but has a bunch of far-ranging product implications (UI changes, may affect presets, etc).

I think one way to explore this is to ask how this API is going to expand in the future? Use cases including:

1. The user returning information about why a run was *not* scheduled
2. The user returning information about a future potential run.

There may be others.

I would also consider how a user would describe what they are returning to their colleagues in an english sentence.

"At every tick a sensor returns instructions back to the system. This can include runs to be launched,"

or some other thing.