This adds a CLI to print the stored versions of each output produced by a pipeline. It can be executed by running `dagster pipeline print_versions <options>`.
ClosedPublic
Actions

Authored by • cdecarolis on Oct 7 2020, 9:54 PM.

Details

Reviewers

• sandyryza
yuhan
alangenfeld

Commits

R1:a1cf6a5862e5: This adds a CLI to print the stored versions of each output produced by a…

Summary

During the memoized development loop, it would be useful for users to see how their changes to a pipeline will change computed versions, and also to visualize which outputs need to be reconstructed. This CLI aims to provide a simplified way to do that.

In order to use it, a user would call dagster pipeline print_versions <options>, where the options are the same as those available from dagster pipeline execute (excluding solid selection and tags).

An example output:

A None address denotes that no address has been stored with the corresponding version.

Test Plan

Added tests in test_memoized_development_cli.py.

Diff Detail

Repository

R1 dagster

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

• cdecarolis created this revision.Oct 7 2020, 9:54 PM

• cdecarolis added reviewers: • sandyryza, yuhan.Oct 7 2020, 9:54 PM

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 7 2020, 10:10 PM

Harbormaster failed remote builds in B19292: Diff 23441!

• cdecarolis added a reviewer: alangenfeld.Oct 8 2020, 3:15 PM

Harbormaster completed remote builds in B19312: Diff 23468.Oct 8 2020, 3:23 PM

• cdecarolis requested review of this revision.Oct 8 2020, 3:23 PM

• cdecarolis edited the summary of this revision. (Show Details)Oct 8 2020, 3:40 PM

• cdecarolis edited the summary of this revision. (Show Details)

Nice! This implementation looks very straightforward. I thought it would need to be more complicated.

What do you think about just calling this versions or output_versions or maybe list_versions? The fact that we'll be printing stuff is kinda implicit in the act that it's a cli command.

python_modules/dagster/dagster/cli/pipeline.py
244	Is this option replicated across "execute", "launch", and this command? Could make sense to define it in one place?
254	Does this documentation apply?
308	This is a nitpick, but it would be a little more concise to do: address = step_output_addresses.get((pipeline_name, step_output_handle), "None") table.append( [ "{key}.{output}".format( key=step_output_handle.step_key, output=step_output_handle.output_name ), version, address, ] )

Changed print_versions -> list_versions. Consolidated config option. Made list_versions logic more concise.

Harbormaster completed remote builds in B19369: Diff 23536.Oct 9 2020, 4:44 PM

This LGTM. As chatted about in Slack, we should get approval in the #proposed-api-changes channel.

This revision is now accepted and ready to land.Oct 13 2020, 1:00 AM

alangenfeld added inline comments.Oct 13 2020, 3:03 PM

python_modules/dagster/setup.py
80	does this have any transitive deps?

• cdecarolis added inline comments.Oct 13 2020, 3:11 PM

python_modules/dagster/setup.py
80	I don't believe so. According to their `setup.py` on github (https://github.com/astanin/python-tabulate/blob/master/setup.py), it doesn't require anything beyond having an actual python install.