Page MenuHomePhabricator

Convert gcp_pipeline to use partitions
ClosedPublic

Authored by sashank on Tue, Jan 14, 1:31 AM.

Details

Summary

Previously, the gcp pipeline's composite solids depended on execution_epoch_time tags. This diff replaces it with a 'date' field in config, and sets up a schedule to datetime partition the pipeline.

Test Plan

load dagit, select different partitions, verify config is correctly generated

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

sashank created this revision.Tue, Jan 14, 1:31 AM
schrockn accepted this revision.Tue, Jan 14, 2:10 AM
This revision is now accepted and ready to land.Tue, Jan 14, 2:10 AM