Page MenuHomePhabricator

[RFC] Alias Builder
AbandonedPublic

Authored by schrockn on Wed, Oct 9, 4:33 AM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Summary

Proposed alternative syntax for specifying aliases

Just getting some feedback here. I think this feels a tad better than
the alias stuff, which is pretty brutal.

Test Plan

BK

Diff Detail

Repository
R1 dagster
Branch
new-alias-syntax
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

schrockn updated this revision to Diff 5727.Wed, Oct 9, 4:33 AM
schrockn created this revision.

up

schrockn updated this revision to Diff 5728.Wed, Oct 9, 4:35 AM
schrockn edited the summary of this revision. (Show Details)
schrockn added reviewers: Restricted Project, alangenfeld.

upmessage

Harbormaster completed remote builds in B4584: Diff 5727.
alangenfeld added inline comments.Wed, Oct 9, 5:44 PM
examples/dagster_examples/airline_demo/pipelines.py
98–101

I wonder if in the scope of a composition function we could see where the colliding solids are used as inputs/outputs and use that to generate the unique suffixes.

@pipeline(
   ...
  autoalias=True
)
def airline_demo_ingest_pipeline():
    load_data_to_database_from_spark( # infer suffix from name of solid used for input join_q2_data
        data_frame=join_q2_data(
            april_data=s3_to_df(), # these would get suffix from what they are hooked up to ie april_data
            may_data=s3_to_df(), 
            june_data=s3_to_df(), 
            master_cord_data=s3_to_df()

It would have to be some pretty complicated multi-round resolution algorithm that ran after assigning them all temporary unique names. Could always bail in complicated cases and leave it as the simple _N suffix.

114

hm cute idea, can't say its worse than before but torn if better. With only .named. in the middle I find it hard to parse the big block of them. Wonder if parens would help that, but thats its own trade off.

s3_to_df.named().april_on_time_s3_to_df()
s3_to_df.as().april_on_time_s3_to_df()

schrockn abandoned this revision.Fri, Oct 11, 7:56 PM

with your new stuff I don't think this is necessary

examples/dagster_examples/airline_demo/pipelines.py
98–101

yeah this sounds complicated. very interesting thought though.

114

unfortunately as is a keyword. otherwise would have used it.

I actually think *not* having the parens is one of values here. It is less intimidating for users who don't understand higher order functions, of which there are many