Page MenuHomeElementl
Feed Advanced Search

Jul 29 2021

sandyryza committed R1:8863ab297f3b: indexed coo matrix in story recommender pipeline (authored by sandyryza).
indexed coo matrix in story recommender pipeline
Jul 29 2021, 11:24 PM
sandyryza updated the diff for D9146: indexed coo matrix in story recommender pipeline.

up

Jul 29 2021, 9:18 PM
sandyryza added a comment to D9074: [caprisun] Run ops after assets.

I thought about it a bit more, and it seems like we already have a concept for "thing that should get run after a node" with SolidHooks. Certainly they are a bit more limited in functionality than Ops (hard to chain together), but considering they have access to the output object of the node they're applied to, as well as the resource system, it seems like they should be expressive enough for any alerting-type behavior, have the benefit of easy reusability, have the correct rerun behavior, and don't require the addition of a new concept. Curious to hear your thoughts.

Jul 29 2021, 9:17 PM
sandyryza updated the summary of D9146: indexed coo matrix in story recommender pipeline.
Jul 29 2021, 8:15 PM
sandyryza updated the diff for D9070: [caprisun] RepositoryDefinition.asset_definition_graph.

up

Jul 29 2021, 8:13 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 29 2021, 8:11 PM
sandyryza requested review of D9146: indexed coo matrix in story recommender pipeline.
Jul 29 2021, 6:49 PM
sandyryza added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Jul 29 2021, 5:17 PM
sandyryza retitled D9070: [caprisun] RepositoryDefinition.asset_definition_graph from RepositoryDefinition.get_asset_definition_graph to RepositoryDefinition.asset_definition_graph.
Jul 29 2021, 5:16 PM
sandyryza retitled D9070: [caprisun] RepositoryDefinition.asset_definition_graph from [RFC] RepositoryDefinition.get_asset_definition_graph to RepositoryDefinition.get_asset_definition_graph.
Jul 29 2021, 5:15 PM
sandyryza updated the diff for D9070: [caprisun] RepositoryDefinition.asset_definition_graph.

up

Jul 29 2021, 5:14 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 29 2021, 5:06 PM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from [RFC] @asset and build_assets_job to @asset and build_assets_job.
Jul 29 2021, 4:43 PM
sandyryza added inline comments to D9070: [caprisun] RepositoryDefinition.asset_definition_graph.
Jul 29 2021, 3:25 PM
sandyryza added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Jul 29 2021, 3:10 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 29 2021, 5:12 AM
sandyryza added a comment to D8969: [caprisun] @asset and build_assets_job.

I added docstrings and beefed up the tests - I think this is ready to go in for others to build on top of. It'll be hidden behind dagster.assets, so still not part of the public API.

Jul 29 2021, 3:29 AM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 29 2021, 3:27 AM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from [RFC] @asset and build_assets_job to @asset and build_assets_job.
Jul 29 2021, 3:26 AM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 29 2021, 3:23 AM

Jul 28 2021

sandyryza added a comment to D9074: [caprisun] Run ops after assets.

I'm wondering if people shouldn't be able to just add nodes to their list of assets, and we infer them as such / foist them into an asset?

Jul 28 2021, 11:40 PM
sandyryza added a comment to D9061: [memoization 4/n][rfc] memoization strategy for jobs.

+1 to having a Strategy class that can version both solids and resources. IMO preferable to call it VersionStrategy or VersioningStrategy, because it's supplying versions, which may or may not be used for memoization.

Jul 28 2021, 11:37 PM
sandyryza closed D9098: add instructions for setting up DAGSTER_HOME to new-project README.
Jul 28 2021, 5:02 PM
sandyryza committed R1:c1cc68a27bc5: add instructions for setting up DAGSTER_HOME to new-project README (authored by sandyryza).
add instructions for setting up DAGSTER_HOME to new-project README
Jul 28 2021, 5:02 PM

Jul 27 2021

sandyryza requested review of D9098: add instructions for setting up DAGSTER_HOME to new-project README.
Jul 27 2021, 9:05 PM
sandyryza added inline comments to D9070: [caprisun] RepositoryDefinition.asset_definition_graph.
Jul 27 2021, 8:59 PM
sandyryza added a comment to D8969: [caprisun] @asset and build_assets_job.

@owen - I updated the diff to include io_manager_keys

Jul 27 2021, 5:50 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 27 2021, 5:21 PM
sandyryza updated the summary of D9074: [caprisun] Run ops after assets.
Jul 27 2021, 4:05 PM
sandyryza requested review of D9074: [caprisun] Run ops after assets.
Jul 27 2021, 3:59 PM
sandyryza requested review of D9070: [caprisun] RepositoryDefinition.asset_definition_graph.
Jul 27 2021, 1:28 AM
sandyryza accepted D9017: [docs] sensor concept include run status sensor.

Hmm maybe I misread and the pipeline failure sensor was already nested under the run status sensor? Accepting!

Jul 27 2021, 1:08 AM
sandyryza accepted D9060: [memoization 5/n] enable memoization with root input managers.
Jul 27 2021, 1:01 AM

Jul 26 2021

sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 26 2021, 11:38 PM
sandyryza updated the diff for D8969: [caprisun] @asset and build_assets_job.

up

Jul 26 2021, 11:35 PM
sandyryza added a comment to D8969: [caprisun] @asset and build_assets_job.

@alangenfeld

One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?

Jul 26 2021, 3:56 PM
sandyryza accepted D8956: [dagit] Show start/end information on latest run in Job details.
Jul 26 2021, 3:51 PM
sandyryza added a comment to D9017: [docs] sensor concept include run status sensor.

Since pipeline failure sensors are the most common case, would it make sense to put them above the run status sensor? Alternatively, would it make sense to nest them inside?

Jul 26 2021, 3:50 PM
sandyryza closed D8946: nothing example misuses type annotations.
Jul 26 2021, 3:44 PM
sandyryza committed R1:da8c5d665cf7: nothing example misuses type annotations (authored by sandyryza).
nothing example misuses type annotations
Jul 26 2021, 3:44 PM
sandyryza closed D8998: check.is_dict.
Jul 26 2021, 3:43 PM
sandyryza committed R1:98ac98c74ac9: check.is_dict (authored by sandyryza).
check.is_dict
Jul 26 2021, 3:43 PM
sandyryza closed D9000: add title and description to graph migration page.
Jul 26 2021, 3:40 PM
sandyryza committed R1:d245b4e3dd54: add title and description to graph migration page (authored by sandyryza).
add title and description to graph migration page
Jul 26 2021, 3:40 PM

Jul 22 2021

sandyryza closed D9021: changes for 0.12.3.
Jul 22 2021, 7:55 PM
sandyryza committed R1:af93a816e25a: changes for 0.12.3 (authored by sandyryza).
changes for 0.12.3
Jul 22 2021, 7:55 PM
sandyryza updated the diff for D9021: changes for 0.12.3.

up

Jul 22 2021, 7:22 PM
sandyryza added a comment to D9021: changes for 0.12.3.

@jordansanders how strongly do you feel about stars vs. dashes? I don't mind changing this time, but fairly annoying if that needs to be another step in the release guide. (the stars are generated by the Quip markdown export).

Jul 22 2021, 7:18 PM
sandyryza published D9021: changes for 0.12.3 for review.
Jul 22 2021, 7:05 PM
sandyryza accepted D9015: run_status_sensor api doc.
Jul 22 2021, 6:04 PM
sandyryza published D9015: run_status_sensor api doc for review.
Jul 22 2021, 6:04 PM
sandyryza accepted D8942: [crag] Job description.
Jul 22 2021, 3:47 PM
sandyryza accepted D8943: [crag] fix snapshots handling composed @graph.
Jul 22 2021, 3:33 PM
sandyryza accepted D8900: RunStatusSensorDefinition and @run_status_sensor.

This looks great

Jul 22 2021, 12:41 AM
sandyryza added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 22 2021, 12:14 AM

Jul 21 2021

sandyryza accepted D8992: [dagit] Fix link to partition from schedule page.
Jul 21 2021, 11:32 PM
sandyryza updated the summary of D8969: [caprisun] @asset and build_assets_job.
Jul 21 2021, 11:32 PM
sandyryza updated the summary of D8969: [caprisun] @asset and build_assets_job.
Jul 21 2021, 11:30 PM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from assets to [RFC] @asset and build_assets_job.
Jul 21 2021, 11:29 PM
sandyryza added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 21 2021, 11:28 PM
sandyryza requested review of D8969: [caprisun] @asset and build_assets_job.
Jul 21 2021, 11:01 PM
sandyryza requested review of D9000: add title and description to graph migration page.
Jul 21 2021, 10:50 PM
sandyryza requested review of D8998: check.is_dict.
Jul 21 2021, 10:29 PM
sandyryza closed D8976: [new-project] sensor targets non-existent mode.
Jul 21 2021, 10:27 PM
sandyryza committed R1:d774b3ff1ca0: [new-project] sensor targets non-existent mode (authored by sandyryza).
[new-project] sensor targets non-existent mode
Jul 21 2021, 10:27 PM
sandyryza accepted D8979: [crag] partition schedules migration guide.

This looks great

Jul 21 2021, 9:48 PM
sandyryza added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

I agree with Alex regarding the opt-in question

Jul 21 2021, 4:34 PM
sandyryza added inline comments to D8979: [crag] partition schedules migration guide.
Jul 21 2021, 12:45 AM

Jul 20 2021

sandyryza requested review of D8976: [new-project] sensor targets non-existent mode.
Jul 20 2021, 11:21 PM
sandyryza added inline comments to D8900: RunStatusSensorDefinition and @run_status_sensor.
Jul 20 2021, 11:02 PM
sandyryza abandoned D8180: [RFC] funchouse.
Jul 20 2021, 5:24 PM
sandyryza added a comment to D8180: [RFC] funchouse.

abandoning this in favor of a version that will live inside dagster core

Jul 20 2021, 5:24 PM
sandyryza added a comment to D8942: [crag] Job description.

Do you have thoughts on what the dagit experience should be like? I.e. where does graph description vs. job description show up?

Jul 20 2021, 3:32 PM
sandyryza accepted D8954: [dagit] Crag: solid -> op.
Jul 20 2021, 3:27 PM
sandyryza resigned from D8501: refactor: convert toys to graph + job + op(eration)s.
Jul 20 2021, 3:24 PM
sandyryza closed D8918: improvements to graph/job/op migration guide.
Jul 20 2021, 3:17 PM
sandyryza committed R1:b372aa199679: improvements to graph/job/op migration guide (authored by sandyryza).
improvements to graph/job/op migration guide
Jul 20 2021, 3:17 PM

Jul 19 2021

sandyryza added a comment to D8956: [dagit] Show start/end information on latest run in Job details.

Thoughts on replacing the duration with the most recent start time in the header? I think the most recent start time is a more useful piece of information than the duration.

Jul 19 2021, 9:34 PM
sandyryza accepted D8949: [dagit] Fix non-Crag schedule/sensor job metadata.
Jul 19 2021, 8:52 PM
sandyryza requested review of D8946: nothing example misuses type annotations.
Jul 19 2021, 5:58 PM
sandyryza accepted D8945: [crag] jobs entry in @repository dict form.
Jul 19 2021, 5:47 PM
sandyryza retitled D8918: improvements to graph/job/op migration guide from resource instance in graph/job/op migration guide to improvements to graph/job/op migration guide.
Jul 19 2021, 4:50 PM
sandyryza requested review of D8918: improvements to graph/job/op migration guide.
Jul 19 2021, 4:47 PM

Jul 16 2021

sandyryza accepted D8903: Add solid as parameter to build_hook_context.
Jul 16 2021, 12:48 AM
sandyryza accepted D8916: put under test that jobs with partitioned config can be successfully backfilled..

great that this already works

Jul 16 2021, 12:46 AM

Jul 15 2021

sandyryza closed D8907: nav entry for the graph/job/op migration guide.
Jul 15 2021, 7:51 PM
sandyryza committed R1:dd6d4ceb2240: nav entry for the graph/job/op migration guide (authored by sandyryza).
nav entry for the graph/job/op migration guide
Jul 15 2021, 7:51 PM
sandyryza updated the diff for D8907: nav entry for the graph/job/op migration guide.

up

Jul 15 2021, 5:35 PM
sandyryza updated the diff for D8907: nav entry for the graph/job/op migration guide.

up

Jul 15 2021, 5:07 PM

Jul 14 2021

sandyryza accepted D8897: [dagit] Move job metadata into header.

Nice

Jul 14 2021, 11:36 PM
sandyryza added a comment to D8900: RunStatusSensorDefinition and @run_status_sensor.

Thoughts on calling this pipeline_event_sensor? pipeline_sensor could be interpreted as a sensor that kicks off a pipeline.

Jul 14 2021, 11:04 PM
sandyryza accepted D8678: [RFC] time-based partitioned config and schedule from partitioned config.

I left a final comment that should be addressed before merging. Otherwise, looks great.

Jul 14 2021, 2:30 AM

Jul 13 2021

sandyryza requested changes to D8678: [RFC] time-based partitioned config and schedule from partitioned config.
Jul 13 2021, 8:47 PM
sandyryza closed D8821: fix error message when solid is missing context.
Jul 13 2021, 5:24 PM
sandyryza committed R1:d206f5e08beb: fix error message when solid is missing context (authored by sandyryza).
fix error message when solid is missing context
Jul 13 2021, 5:24 PM
sandyryza requested changes to D8565: [crag] permit resource instances on to_job.

Oops accepted by accident

Jul 13 2021, 5:24 PM
sandyryza accepted D8565: [crag] permit resource instances on to_job.

I had originally been a strong proponent of this, but, having reviewed the resources in our dogfood pipelines, I'm less sure. It's only really the IOManagers that would benefit from this. I wonder if it would make more sense to just make it a little less verbose to make a hardcoded resource. Right now it requires ResourceDefinition.hardcoded_resource. We could make hardcoded_resource a top-level API?

Jul 13 2021, 5:23 PM
sandyryza added a comment to D8678: [RFC] time-based partitioned config and schedule from partitioned config.

Thanks a ton for picking this up. To make it accessible to users, it needs a few more things:

  • Docstrings for the _partitioned_config functions.
  • Experimental warnings.
  • Inclusion in the experimental API docs page.
  • Inclusion in the top-level public API.
Jul 13 2021, 5:14 PM

Jul 12 2021

sandyryza closed D8824: crag migration guide.
Jul 12 2021, 11:27 PM
sandyryza committed R1:11a266443fdf: crag migration guide (authored by sandyryza).
crag migration guide
Jul 12 2021, 11:27 PM