Page MenuHomeElementl

Add an example of how to write a user-in-the-loop pipeline
ClosedPublic

Authored by yichendai on Feb 18 2021, 4:15 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Mar 26, 12:38 AM
Unknown Object (File)
Fri, Mar 24, 8:49 PM
Unknown Object (File)
Dec 30 2022, 1:47 PM
Unknown Object (File)
Dec 30 2022, 12:27 AM
Unknown Object (File)
Dec 27 2022, 10:04 AM
Unknown Object (File)
Dec 27 2022, 4:49 AM
Unknown Object (File)
Dec 27 2022, 3:19 AM
Unknown Object (File)
Dec 25 2022, 10:18 PM
Subscribers

Details

Summary

This diff adds an example of how to write pipelines to include a user in the loop.

The documentation is added.

The description for some examples has a period at the end, others hasn't. To be consistent, I also modified those description.

Test Plan

See tests

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 18 2021, 4:35 AM
Harbormaster failed remote builds in B26080: Diff 31837!
yichendai retitled this revision from Add an example for user in the loop. Add tests. to Add an example of how to write a user-in-the-loop pipeline.Feb 18 2021, 4:43 PM
examples/user_in_loop/logger.yaml
2 ↗(On Diff #31837)

let's drop this custom logger

examples/user_in_loop/repo.py
18 ↗(On Diff #31837)

let's drop this custom logger

59 ↗(On Diff #31837)

we can't hardcode this path -- it needs to be set in config

Parameterizing solid with run config and delete custom logger

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2021, 1:03 AM
Harbormaster failed remote builds in B26308: Diff 32146!
Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2021, 7:40 AM
Harbormaster failed remote builds in B26317: Diff 32158!
examples/user_in_loop/tests/test_user_in_loop_pipeline.py
26

One thing I'm not sure here is if this is a good way to create a temporary file for testing.

I do have another command at the bottom (line 60) to remove it after the test.

examples/user_in_loop/tests/test_user_in_loop_pipeline.py
26
examples/user_in_loop/tests/test_user_in_loop_pipeline.py
26

Or is it better to use a tempfile module ?

yichendai added inline comments.
examples/user_in_loop/tests/test_user_in_loop_pipeline.py
26

Cool! thanks!

yichendai marked an inline comment as done.

Use temp dir to remove created file

max requested changes to this revision.Feb 24 2021, 5:37 PM
max added inline comments.
examples/user_in_loop/tests/test_user_in_loop_pipeline.py
26

yes you should use tempfile

38

better to have descriptive names here

examples/user_in_loop/user_in_loop/repo.py
38

i think we should simplify this pipeline down to be a single solid. among other things, nothing actually blocks on wait_for_user_approval

This revision now requires changes to proceed.Feb 24 2021, 5:37 PM
examples/user_in_loop/user_in_loop/repo.py
33

this should be much faster since it's going to be tested

examples/user_in_loop/user_in_loop/repo.py
25

consider sentinel_file_path

28

file_path or sentinel_file_path

29

clearer to use os.path.exists

examples/user_in_loop/user_in_loop/repo.py
29

Make sense. Will do!

38

haha, that's true.

Simplify the pipeline and docs

max added inline comments.
examples/user_in_loop/tests/test_user_in_loop_pipeline.py
27

if you do things this way, you're assuming that touch is available on the user's computer (this isn't true, e.g., on windows). instead:

with open(file_path, 'r') as fd:
    fd.write('ok')

(or use dagster.utils.touch_file)

examples/user_in_loop/user_in_loop/repo.py
12

we can drop the business logic completely (what is a price list? what is this price list? what does it have to do with anything) -- just leave a log line at the end of the solid

This revision is now accepted and ready to land.Feb 24 2021, 9:47 PM
examples/user_in_loop/tests/test_user_in_loop_pipeline.py
27

Gotta. Didn't recall the way of create a file using writing a file 😮will update it!

yichendai marked an inline comment as done.

Simplify the solid and docs completely. Use open file rather than subprocess to create a file.