Page MenuHomeElementl

[Memoized Dev Example 2/2] Memoized development example
ClosedPublic

Authored by cdecarolis on Jan 19 2021, 10:28 PM.

Details

Summary

This diff adds a code example / tutorial for versioning/memoization. It is intended to get someone started with memoization in a relatively simple way, but also assumes some background knowledge about dagster concepts.

Test Plan

Screen Shot 2021-01-19 at 5.25.57 PM.png (1×1 px, 269 KB)

Screen Shot 2021-01-19 at 5.26.06 PM.png (952×1 px, 145 KB)

Screen Shot 2021-01-19 at 5.26.20 PM.png (1×1 px, 207 KB)

Screen Shot 2021-01-19 at 5.26.29 PM.png (900×1 px, 187 KB)

Screen Shot 2021-01-19 at 5.26.38 PM.png (825×270 px, 57 KB)

Test docs build and docs unit tests.

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 19 2021, 11:02 PM
Harbormaster failed remote builds in B24553: Diff 29898!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 20 2021, 3:31 PM
Harbormaster failed remote builds in B24569: Diff 29920!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 20 2021, 4:17 PM
Harbormaster failed remote builds in B24571: Diff 29924!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 20 2021, 4:43 PM
Harbormaster failed remote builds in B24576: Diff 29929!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 7:43 PM
Harbormaster failed remote builds in B24654: Diff 30022!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 9:05 PM
Harbormaster failed remote builds in B24661: Diff 30031!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 9:55 PM
Harbormaster failed remote builds in B24667: Diff 30037!

Forgot to install memoized dev as its own package

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 10:31 PM
Harbormaster failed remote builds in B24670: Diff 30045!
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 22 2021, 3:58 PM
Harbormaster failed remote builds in B24708: Diff 30089!

others have more context on this than I

docs/next/src/pages/examples/memoized_development.mdx
5

can include <ExampleReferenceLink filePath="examples/memoized_development" /> so you will get the little github link widget.

25

maybe we can highlight the line tags={MEMOIZED_RUN_TAG: "true"},

34

should we link versioned_filesystem_io_manager to the api doc (using PyObject)?

35

this breaks the line.

image.png (170×1 px, 35 KB)

47

nit: python -> Python

48–50

it may not be very clear to the reader if this get_hash_for_file is system-provided or needs to be implemented by themselves.

61

caption=sentence.py?

docs/next/src/pages/examples/memoized_development.mdx
48–50

good point. I guess I'll clarify that this needs to be provided by them, but we've written a basic implementation here.

61

ah good catch thanks.

cdecarolis marked 7 inline comments as done.

Addressed comments

looking great! thanks for putting this together. very much look forward to getting this in front of users.

This revision is now accepted and ready to land.Jan 25 2021, 11:49 PM