Page MenuHomeElementl
Feed Advanced Search

Yesterday

sandyryza closed D9195: add composite solids to graph/job/op migration guide.
Wed, Aug 4, 11:44 PM
sandyryza committed R1:5b279196a578: add composite solids to graph/job/op migration guide (authored by sandyryza).
add composite solids to graph/job/op migration guide
Wed, Aug 4, 11:44 PM
sandyryza added a comment to D9195: add composite solids to graph/job/op migration guide.

@alangenfeld good point - I updated to give the graphs inputs and outputs

Wed, Aug 4, 7:02 PM
sandyryza added inline comments to D9195: add composite solids to graph/job/op migration guide.
Wed, Aug 4, 7:01 PM
sandyryza updated the diff for D9195: add composite solids to graph/job/op migration guide.

up

Wed, Aug 4, 6:14 PM
sandyryza updated the diff for D9195: add composite solids to graph/job/op migration guide.

up

Wed, Aug 4, 4:40 PM
sandyryza closed D9205: [demo pipelines] remove slack on success hook.
Wed, Aug 4, 4:30 PM
sandyryza committed R1:4c68172cfcd0: [demo pipelines] remove slack on success hook (authored by sandyryza).
[demo pipelines] remove slack on success hook
Wed, Aug 4, 4:30 PM
sandyryza requested review of D9205: [demo pipelines] remove slack on success hook.
Wed, Aug 4, 3:57 PM
sandyryza added a comment to D9203: WIP: Dockerfile/deployment instructions for deploying hacker-news to k8s.

Thanks a ton for putting this together @dgibson!

Wed, Aug 4, 3:48 PM
sandyryza closed D9201: [crag] add GraphDefinition to the public API.
Wed, Aug 4, 3:11 PM
sandyryza committed R1:56cb2f549956: [crag] add GraphDefinition to the public API (authored by sandyryza).
[crag] add GraphDefinition to the public API
Wed, Aug 4, 3:11 PM

Tue, Aug 3

sandyryza requested review of D9201: [crag] add GraphDefinition to the public API.
Tue, Aug 3, 7:15 PM
sandyryza requested review of D9200: [crag] missing api references in graph/job/op migration guide.
Tue, Aug 3, 7:12 PM
sandyryza closed D9172: [crag] remove job arg from partitioned schedule decorators.
Tue, Aug 3, 7:07 PM
sandyryza committed R1:b064af1e2f94: [crag] remove job arg from partitioned schedule decorators (authored by sandyryza).
[crag] remove job arg from partitioned schedule decorators
Tue, Aug 3, 7:06 PM
sandyryza added inline comments to D9172: [crag] remove job arg from partitioned schedule decorators.
Tue, Aug 3, 7:06 PM
sandyryza updated the diff for D9172: [crag] remove job arg from partitioned schedule decorators.

up

Tue, Aug 3, 6:09 PM
sandyryza requested review of D9195: add composite solids to graph/job/op migration guide.
Tue, Aug 3, 4:33 PM
sandyryza updated the diff for D9172: [crag] remove job arg from partitioned schedule decorators.

up

Tue, Aug 3, 4:27 PM
sandyryza retitled D9187: [crag] use non-Nothing inputs in crag toys from use non-Nothing inputs in crag toys to [crag] use non-Nothing inputs in crag toys.
Tue, Aug 3, 4:10 PM
sandyryza updated the diff for D9156: [caprisun] asset in.

up

Tue, Aug 3, 4:09 PM
sandyryza accepted D9061: [memoization 4/n][rfc] memoization strategy for jobs.

I like it

Tue, Aug 3, 1:26 AM

Mon, Aug 2

sandyryza added reviewers for D9188: [caprisun] use @asset in the story recommender pipeline: cdecarolis, bengotow, prha.
Mon, Aug 2, 10:49 PM
sandyryza requested review of D9188: [caprisun] use @asset in the story recommender pipeline.
Mon, Aug 2, 10:46 PM
sandyryza added reviewers for D9187: [crag] use non-Nothing inputs in crag toys: bengotow, cdecarolis.
Mon, Aug 2, 10:27 PM
sandyryza updated the diff for D9156: [caprisun] asset in.

up

Mon, Aug 2, 10:12 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Mon, Aug 2, 10:09 PM
sandyryza requested review of D9187: [crag] use non-Nothing inputs in crag toys.
Mon, Aug 2, 9:55 PM
sandyryza closed D9184: split up test_decorators.
Mon, Aug 2, 9:34 PM
sandyryza committed R1:72ba29df076e: split up test_decorators (authored by sandyryza).
split up test_decorators
Mon, Aug 2, 9:34 PM
sandyryza closed D9171: [crag] allow non-string tag values in to_job.
Mon, Aug 2, 8:25 PM
sandyryza committed R1:1f936709ec27: [crag] allow non-string tag values in to_job (authored by sandyryza).
[crag] allow non-string tag values in to_job
Mon, Aug 2, 8:24 PM
sandyryza retitled D9171: [crag] allow non-string tag values in to_job from allow non-string tag values in to_job to [crag] allow non-string tag values in to_job.
Mon, Aug 2, 8:23 PM
sandyryza retitled D9173: [crag] [RFC] accept job as a string when constructing schedules from [RFC] accept job as a string when constructing schedules to [crag] [RFC] accept job as a string when constructing schedules.
Mon, Aug 2, 8:22 PM
sandyryza retitled D9172: [crag] remove job arg from partitioned schedule decorators from remove job arg from partitioned schedule decorators to [crag] remove job arg from partitioned schedule decorators.
Mon, Aug 2, 8:21 PM
sandyryza retitled D7722: [crag] cragitize the IO manager concepts page from cragitize the IO manager concepts page to [crag] cragitize the IO manager concepts page.
Mon, Aug 2, 8:21 PM
sandyryza retitled D9070: [caprisun] RepositoryDefinition.asset_definition_graph from RepositoryDefinition.asset_definition_graph to [caprisun] RepositoryDefinition.asset_definition_graph.
Mon, Aug 2, 8:20 PM
sandyryza retitled D9074: [caprisun] Run ops after assets from Run ops after assets to [caprisun] Run ops after assets.
Mon, Aug 2, 8:20 PM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from @asset and build_assets_job to [caprisun] @asset and build_assets_job.
Mon, Aug 2, 8:14 PM
sandyryza retitled D9156: [caprisun] asset in from asset in to [caprisun] asset in.
Mon, Aug 2, 8:00 PM
sandyryza requested review of D9172: [crag] remove job arg from partitioned schedule decorators.
Mon, Aug 2, 5:07 PM
sandyryza requested review of D9184: split up test_decorators.
Mon, Aug 2, 4:56 PM
sandyryza requested review of D9173: [crag] [RFC] accept job as a string when constructing schedules.
Mon, Aug 2, 3:48 PM
sandyryza closed D9174: type check failures often include two adjacent periods.
Mon, Aug 2, 3:47 PM
sandyryza committed R1:5e59e859a2cd: type check failures often include two adjacent periods (authored by sandyryza).
type check failures often include two adjacent periods
Mon, Aug 2, 3:47 PM
sandyryza updated the diff for D9171: [crag] allow non-string tag values in to_job.

up

Mon, Aug 2, 3:39 PM

Sat, Jul 31

sandyryza requested review of D9171: [crag] allow non-string tag values in to_job.
Sat, Jul 31, 5:32 PM
sandyryza updated the diff for D9174: type check failures often include two adjacent periods.

up

Sat, Jul 31, 1:12 AM
sandyryza updated the diff for D9174: type check failures often include two adjacent periods.

up

Sat, Jul 31, 1:10 AM
sandyryza requested review of D9174: type check failures often include two adjacent periods.
Sat, Jul 31, 12:00 AM

Fri, Jul 30

sandyryza accepted D9085: [memoization 8/n] make default fs io manager memoizable.

I think this makes sense. I feel like there might be an edge case out there where this causes trouble, but having trouble thinking of it. With this, are we able to get rid of VersionedPickledObjectFilesystemIOManager?

Fri, Jul 30, 3:17 PM
sandyryza accepted D9154: Adding hacker_news_dbt folder.
Fri, Jul 30, 3:11 PM
sandyryza resigned from D8981: add `from_failure` bool arg to rexecute pipeline apis.

I'm going to defer to others on this

Fri, Jul 30, 3:11 PM

Thu, Jul 29

sandyryza requested review of D9156: [caprisun] asset in.
Thu, Jul 29, 11:54 PM
sandyryza closed D9146: indexed coo matrix in story recommender pipeline.
Thu, Jul 29, 11:24 PM
sandyryza committed R1:8863ab297f3b: indexed coo matrix in story recommender pipeline (authored by sandyryza).
indexed coo matrix in story recommender pipeline
Thu, Jul 29, 11:24 PM
sandyryza updated the diff for D9146: indexed coo matrix in story recommender pipeline.

up

Thu, Jul 29, 9:18 PM
sandyryza added a comment to D9074: [caprisun] Run ops after assets.

I thought about it a bit more, and it seems like we already have a concept for "thing that should get run after a node" with SolidHooks. Certainly they are a bit more limited in functionality than Ops (hard to chain together), but considering they have access to the output object of the node they're applied to, as well as the resource system, it seems like they should be expressive enough for any alerting-type behavior, have the benefit of easy reusability, have the correct rerun behavior, and don't require the addition of a new concept. Curious to hear your thoughts.

Thu, Jul 29, 9:17 PM
sandyryza updated the summary of D9146: indexed coo matrix in story recommender pipeline.
Thu, Jul 29, 8:15 PM
sandyryza updated the diff for D9070: [caprisun] RepositoryDefinition.asset_definition_graph.

up

Thu, Jul 29, 8:13 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Thu, Jul 29, 8:11 PM
sandyryza requested review of D9146: indexed coo matrix in story recommender pipeline.
Thu, Jul 29, 6:49 PM
sandyryza added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Thu, Jul 29, 5:17 PM
sandyryza retitled D9070: [caprisun] RepositoryDefinition.asset_definition_graph from RepositoryDefinition.get_asset_definition_graph to RepositoryDefinition.asset_definition_graph.
Thu, Jul 29, 5:16 PM
sandyryza retitled D9070: [caprisun] RepositoryDefinition.asset_definition_graph from [RFC] RepositoryDefinition.get_asset_definition_graph to RepositoryDefinition.get_asset_definition_graph.
Thu, Jul 29, 5:15 PM
sandyryza updated the diff for D9070: [caprisun] RepositoryDefinition.asset_definition_graph.

up

Thu, Jul 29, 5:14 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Thu, Jul 29, 5:06 PM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from [RFC] @asset and build_assets_job to @asset and build_assets_job.
Thu, Jul 29, 4:43 PM
sandyryza added inline comments to D9070: [caprisun] RepositoryDefinition.asset_definition_graph.
Thu, Jul 29, 3:25 PM
sandyryza added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Thu, Jul 29, 3:10 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Thu, Jul 29, 5:12 AM
sandyryza added a comment to D8969: [caprisun] @asset and build_assets_job.

I added docstrings and beefed up the tests - I think this is ready to go in for others to build on top of. It'll be hidden behind dagster.assets, so still not part of the public API.

Thu, Jul 29, 3:29 AM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Thu, Jul 29, 3:27 AM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from [RFC] @asset and build_assets_job to @asset and build_assets_job.
Thu, Jul 29, 3:26 AM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Thu, Jul 29, 3:23 AM

Wed, Jul 28

sandyryza added a comment to D9074: [caprisun] Run ops after assets.

I'm wondering if people shouldn't be able to just add nodes to their list of assets, and we infer them as such / foist them into an asset?

Wed, Jul 28, 11:40 PM
sandyryza added a comment to D9061: [memoization 4/n][rfc] memoization strategy for jobs.

+1 to having a Strategy class that can version both solids and resources. IMO preferable to call it VersionStrategy or VersioningStrategy, because it's supplying versions, which may or may not be used for memoization.

Wed, Jul 28, 11:37 PM
sandyryza closed D9098: add instructions for setting up DAGSTER_HOME to new-project README.
Wed, Jul 28, 5:02 PM
sandyryza committed R1:c1cc68a27bc5: add instructions for setting up DAGSTER_HOME to new-project README (authored by sandyryza).
add instructions for setting up DAGSTER_HOME to new-project README
Wed, Jul 28, 5:02 PM

Tue, Jul 27

sandyryza requested review of D9098: add instructions for setting up DAGSTER_HOME to new-project README.
Tue, Jul 27, 9:05 PM
sandyryza added inline comments to D9070: [caprisun] RepositoryDefinition.asset_definition_graph.
Tue, Jul 27, 8:59 PM
sandyryza added a comment to D8969: [caprisun] @asset and build_assets_job.

@owen - I updated the diff to include io_manager_keys

Tue, Jul 27, 5:50 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Tue, Jul 27, 5:21 PM
sandyryza updated the summary of D9074: [caprisun] Run ops after assets.
Tue, Jul 27, 4:05 PM
sandyryza requested review of D9074: [caprisun] Run ops after assets.
Tue, Jul 27, 3:59 PM
sandyryza requested review of D9070: [caprisun] RepositoryDefinition.asset_definition_graph.
Tue, Jul 27, 1:28 AM
sandyryza accepted D9017: [docs] sensor concept include run status sensor.

Hmm maybe I misread and the pipeline failure sensor was already nested under the run status sensor? Accepting!

Tue, Jul 27, 1:08 AM
sandyryza accepted D9060: [memoization 5/n] enable memoization with root input managers.
Tue, Jul 27, 1:01 AM

Mon, Jul 26

sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Mon, Jul 26, 11:38 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Mon, Jul 26, 11:35 PM
sandyryza added a comment to D8969: [caprisun] @asset and build_assets_job.

@alangenfeld One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?

Mon, Jul 26, 3:56 PM
sandyryza accepted D8956: [dagit] Show start/end information on latest run in Job details.
Mon, Jul 26, 3:51 PM
sandyryza added a comment to D9017: [docs] sensor concept include run status sensor.

Since pipeline failure sensors are the most common case, would it make sense to put them above the run status sensor? Alternatively, would it make sense to nest them inside?

Mon, Jul 26, 3:50 PM
sandyryza closed D8946: nothing example misuses type annotations.
Mon, Jul 26, 3:44 PM
sandyryza committed R1:da8c5d665cf7: nothing example misuses type annotations (authored by sandyryza).
nothing example misuses type annotations
Mon, Jul 26, 3:44 PM
sandyryza closed D8998: check.is_dict.
Mon, Jul 26, 3:43 PM
sandyryza committed R1:98ac98c74ac9: check.is_dict (authored by sandyryza).
check.is_dict
Mon, Jul 26, 3:43 PM
sandyryza closed D9000: add title and description to graph migration page.
Mon, Jul 26, 3:40 PM
sandyryza committed R1:d245b4e3dd54: add title and description to graph migration page (authored by sandyryza).
add title and description to graph migration page
Mon, Jul 26, 3:40 PM