Page MenuHomePhabricator

Convert gcp_pipeline to use partitions
ClosedPublic

Authored by sashank on Jan 14 2020, 1:31 AM.

Details

Summary

Previously, the gcp pipeline's composite solids depended on execution_epoch_time tags. This diff replaces it with a 'date' field in config, and sets up a schedule to datetime partition the pipeline.

Test Plan

load dagit, select different partitions, verify config is correctly generated

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

sashank created this revision.Jan 14 2020, 1:31 AM
schrockn accepted this revision.Jan 14 2020, 2:10 AM
This revision is now accepted and ready to land.Jan 14 2020, 2:10 AM
This revision was automatically updated to reflect the committed changes.