Page MenuHomePhabricator

This adds a CLI to print the stored versions of each output produced by a pipeline. It can be executed by running `dagster pipeline print_versions <options>`.
ClosedPublic

Authored by cdecarolis on Oct 7 2020, 9:54 PM.

Details

Summary

During the memoized development loop, it would be useful for users to see how their changes to a pipeline will change computed versions, and also to visualize which outputs need to be reconstructed. This CLI aims to provide a simplified way to do that.

In order to use it, a user would call dagster pipeline print_versions <options>, where the options are the same as those available from dagster pipeline execute (excluding solid selection and tags).

An example output:

A None address denotes that no address has been stored with the corresponding version.

Test Plan

Added tests in test_memoized_development_cli.py.

Diff Detail

Repository
R1 dagster
Branch
display_cli
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 7 2020, 10:10 PM
Harbormaster failed remote builds in B19292: Diff 23441!
cdecarolis edited the summary of this revision. (Show Details)

Nice! This implementation looks very straightforward. I thought it would need to be more complicated.

What do you think about just calling this versions or output_versions or maybe list_versions? The fact that we'll be printing stuff is kinda implicit in the act that it's a cli command.

python_modules/dagster/dagster/cli/pipeline.py
243

Is this option replicated across "execute", "launch", and this command? Could make sense to define it in one place?

253

Does this documentation apply?

307

This is a nitpick, but it would be a little more concise to do:

address = step_output_addresses.get((pipeline_name, step_output_handle), "None")
table.append(
    [
        "{key}.{output}".format(
            key=step_output_handle.step_key, output=step_output_handle.output_name
        ),
        version,
        address,
    ]
)

Changed print_versions -> list_versions. Consolidated config option. Made list_versions logic more concise.

This LGTM. As chatted about in Slack, we should get approval in the #proposed-api-changes channel.

This revision is now accepted and ready to land.Oct 13 2020, 1:00 AM
python_modules/dagster/setup.py
80

does this have any transitive deps?

python_modules/dagster/setup.py
80

I don't believe so. According to their setup.py on github (https://github.com/astanin/python-tabulate/blob/master/setup.py), it doesn't require anything beyond having an actual python install.

python_modules/dagster/setup.py
80

thumbsup