Page MenuHomePhabricator

allow importing dagster_aws.emr without transitively importing pyspark
ClosedPublic

Authored by sandyryza on Oct 28 2020, 4:35 PM.

Details

Summary

This makes it possible to import dagster_aws.emr without transitively importing pyspark stuff.

It accomplishes this by making parse_spark_configs generate a configs_spark.py in both the
dagster-aws and dagster-spark packages.

While this is probably not how we'd do it if we were starting over, but it avoids a breaking change.

Depends on D4940

Test Plan

bk + regenerated configs_spark.pys

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

sandyryza retitled this revision from Summary: This reverts the creation of the dagster-aws-pyspark package and instead makes it possible to import dagster_aws.emr without transitively importing pyspark stuff. to Summary:This reverts the creation of the dagster-aws-pyspark package and instead makes it possibleto import dagster_aws.emr without transitively importing pyspark stuff..Oct 28 2020, 4:40 PM
sandyryza edited the summary of this revision. (Show Details)
sandyryza retitled this revision from Summary:This reverts the creation of the dagster-aws-pyspark package and instead makes it possibleto import dagster_aws.emr without transitively importing pyspark stuff. to import dagster_aws.emr without transitively importing pyspark.
sandyryza edited the summary of this revision. (Show Details)
sandyryza retitled this revision from import dagster_aws.emr without transitively importing pyspark to allow importing dagster_aws.emr without transitively importing pyspark.Oct 29 2020, 12:26 AM

I would consider a unit test that ensures that the contents of the two duplicated files are identical,

python_modules/automation/automation/parse_spark_configs.py
41

what's this about?

This revision is now accepted and ready to land.Oct 29 2020, 1:15 AM
This revision was landed with ongoing or failed builds.Oct 29 2020, 7:01 PM
This revision was automatically updated to reflect the committed changes.