Page MenuHomeElementl
Feed Advanced Search

Aug 17 2021

owen accepted D9226: Input/OutputContext.asset_key.
Aug 17 2021, 3:45 PM

Aug 9 2021

owen accepted D9215: [demo pipelines] remove dynamic download pipeline.

makes sense -- I only kept it around in case we ended up using it for something, but I suppose git history will do.

Aug 9 2021, 11:29 PM
owen accepted D9214: [demo pipelines] remove unneeded pylint disable.
Aug 9 2021, 11:28 PM

Aug 4 2021

owen accepted D9205: [demo pipelines] remove slack on success hook.

totally agree with getting rid of this -- it'll also make the dogfooding alert channel much less annoying :)

Aug 4 2021, 4:14 PM

Aug 2 2021

owen accepted D9156: [caprisun] asset in.

Like the general shape of this + how it makes the interface feel more similar to @op

Aug 2 2021, 10:50 PM

Jul 30 2021

owen added a comment to D9074: [caprisun] Run ops after assets.

There are a couple things I'm worried are missing from solid hooks:

  • Re-execution and retries. E.g. imagine you refresh an asset that's a table with a set of emails you want to send, and then you want to send the emails. If the email-sending code fails to connect to the email-sending service, you'd want to be able to re-execute that step without also re-executing the step that generates the table.
  • Executing in separate processes. Generating a table might require running a Spark job on a special Spark cluster or doing inference on a GPU. If you want to do a time-consuming operation afterwards (like send a bunch of emails) that doesn't need the cluster or GPU, it would be wasteful to use the same process.
Jul 30 2021, 4:37 PM
owen closed D9154: Adding hacker_news_dbt folder.
Jul 30 2021, 3:28 PM
owen committed R1:08918daf3375: Adding hacker_news_dbt folder (authored by owen).
Adding hacker_news_dbt folder
Jul 30 2021, 3:28 PM

Jul 29 2021

owen published D9154: Adding hacker_news_dbt folder for review.
Jul 29 2021, 11:08 PM
owen accepted D9146: indexed coo matrix in story recommender pipeline.
Jul 29 2021, 9:09 PM
owen added a comment to D9074: [caprisun] Run ops after assets.

@sandyryza Writing out / formalizing some stuff from our conversation yesterday, it feels like there are two distinct cases where a person might want to include things that don't produce "assets" (long-living objects external to the dagster ecosystem) within their asset oriented pipeline.

Jul 29 2021, 6:30 PM
owen closed D9145: Update readme for hackernews example.
Jul 29 2021, 5:42 PM
owen committed R1:3bde5e02e119: Update readme for hackernews example (authored by owen).
Update readme for hackernews example
Jul 29 2021, 5:42 PM
owen requested review of D9145: Update readme for hackernews example.
Jul 29 2021, 5:22 PM

Jul 28 2021

owen closed D9121: Smarter dbt default argument handling.
Jul 28 2021, 5:51 PM
owen committed R1:7e415872376e: Smarter dbt default argument handling (authored by owen).
Smarter dbt default argument handling
Jul 28 2021, 5:51 PM
owen closed D9103: Add missing imports to hacker_news example repo.py.
Jul 28 2021, 5:24 PM
owen committed R1:6ffd116028b6: Add missing imports to hacker_news example repo.py (authored by owen).
Add missing imports to hacker_news example repo.py
Jul 28 2021, 5:24 PM
owen requested review of D9121: Smarter dbt default argument handling.
Jul 28 2021, 4:43 PM

Jul 27 2021

owen requested review of D9103: Add missing imports to hacker_news example repo.py.
Jul 27 2021, 10:51 PM

Jul 26 2021

owen updated subscribers of D8969: [caprisun] @asset and build_assets_job.

@alangenfeld

One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?

I think this is a question even in the non-@asset world of solids and IOManagers. Our current recommendation is that, in their IOManager, users switch on the Python type attached to the InputDefinition. Here's an example in the hacker news pipeline: https://github.com/dagster-io/dagster/blob/master/examples/hacker_news/hacker_news/resources/parquet_io_manager.py#L46.

Jul 26 2021, 9:23 PM
owen closed D9013: Add dbt pipeline to example repo, synchronize with internal.
Jul 26 2021, 4:20 PM
owen committed R1:4b287b299cae: Add dbt pipeline to example repo, synchronize with internal (authored by owen).
Add dbt pipeline to example repo, synchronize with internal
Jul 26 2021, 4:20 PM

Jul 23 2021

owen published D9013: Add dbt pipeline to example repo, synchronize with internal for review.
Jul 23 2021, 6:51 PM

Jul 21 2021

owen accepted D8990: pull description from doc string on all decorators.

not sure what the deal is with pokemon in reviews but when in rome...

Jul 21 2021, 5:03 PM

Jul 20 2021

owen closed D8939: Better tests for dbt integration guide, and slightly modify example.
Jul 20 2021, 9:00 PM
owen committed R1:f36963e133ac: Better tests for dbt integration guide, and slightly modify example (authored by owen).
Better tests for dbt integration guide, and slightly modify example
Jul 20 2021, 9:00 PM
owen requested review of D8939: Better tests for dbt integration guide, and slightly modify example.
Jul 20 2021, 6:19 PM

Jul 19 2021

owen accepted D8946: nothing example misuses type annotations.
Jul 19 2021, 8:18 PM

Jul 16 2021

owen accepted D8933: NodeHandle serdes adjustments.
Jul 16 2021, 10:30 PM
owen added a comment to D8933: NodeHandle serdes adjustments.
Jul 16 2021, 10:11 PM
owen accepted D8926: PipelineDefinition.all_solid_defs -> all_node_defs.

Macro thumbsup:

Jul 16 2021, 5:37 PM

Jul 14 2021

owen accepted D8895: GraphDefinition.solid_dict -> node_dict.

Macro thumbsup:

Jul 14 2021, 9:40 PM
owen accepted D8715: Solid -> Node.

Nice -- one step closer to banishing solids from this world

Jul 14 2021, 5:49 PM
owen accepted D8877: asset catalog perf: use wipe_timestamp for asset_key filtering.

Macro thumbsup:

Jul 14 2021, 4:14 PM

Jul 13 2021

owen added inline comments to D8876: mention positionals in Nothing example.
Jul 13 2021, 11:06 PM
owen accepted D8876: mention positionals in Nothing example.
Jul 13 2021, 11:04 PM
owen accepted D8864: asset catalog perf: make use of tags column if migrated.
Jul 13 2021, 7:26 PM

Jul 12 2021

owen accepted D8839: Revert "pin dbt for test", try removing default dbt profile dir.
Jul 12 2021, 10:09 PM
owen accepted D8831: pin dbt for test.
Jul 12 2021, 8:39 PM
owen added a comment to D8831: pin dbt for test.

one other option that might work would be to remove the PROFILES_DIR configuration (https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/dbt_example/dbt_example/solids.py?L11). It seems like the value that it's set to is the default value for dbt, so it doesn't really do anything, and might be having some weird interaction with the update. The changelog for dbt mentions some changes to how they interpret the profiles-dir path (relative to the project-dir path vs absolute) that I wouldn't think would impact us, but it seems possible that this would be a buggy side effect

Jul 12 2021, 8:39 PM

Jul 6 2021

owen closed D8691: Add explanation differentiating python types and dagster types.
Jul 6 2021, 11:54 PM
owen committed R1:0a8ea6c5ea10: Add explanation differentiating python types and dagster types (authored by owen).
Add explanation differentiating python types and dagster types
Jul 6 2021, 11:54 PM
owen added a comment to D8691: Add explanation differentiating python types and dagster types.
In D8691#227443, @yuhan wrote:

This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types

Jul 6 2021, 11:07 PM
owen updated the diff for D8691: Add explanation differentiating python types and dagster types.
  • add extra explanation
Jul 6 2021, 11:06 PM
owen added a reviewer for D8691: Add explanation differentiating python types and dagster types: yuhan.
Jul 6 2021, 10:26 PM
owen updated the diff for D8691: Add explanation differentiating python types and dagster types.
  • rebase
Jul 6 2021, 8:17 PM
owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.
  • rebase
Jul 6 2021, 7:38 PM
owen added inline comments to D8392: Refactor dagster-dbt integration / add dbt cli resource.
Jul 6 2021, 6:53 PM
owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.
  • Updated docs for dbt integration
Jul 6 2021, 6:49 PM
owen requested review of D8691: Add explanation differentiating python types and dagster types.
Jul 6 2021, 4:44 PM

Jul 1 2021

owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.
  • up
Jul 1 2021, 11:17 PM
owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.
  • rebase
Jul 1 2021, 10:22 PM
owen accepted D8663: [mypy] pipeline & solid snapshots.

Macro absolute-win:

Jul 1 2021, 10:17 PM
owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.
  • major refactor
  • make black
Jul 1 2021, 10:12 PM
owen added inline comments to D8392: Refactor dagster-dbt integration / add dbt cli resource.
Jul 1 2021, 10:02 PM
owen added inline comments to D8392: Refactor dagster-dbt integration / add dbt cli resource.
Jul 1 2021, 9:59 PM
owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.
  • major refactor
Jul 1 2021, 9:55 PM

Jun 25 2021

owen accepted D8533: [dynamic] multi out map test.
Jun 25 2021, 4:38 PM

Jun 16 2021

owen added a comment to D8387: single asset sensor definition.

Overall, I like this direction, but I do think there are many cases where you might want to depend on multiple asset keys at once (the story_recommender pipeline is one example). This does add bits of complexity basically everywhere (need a fancier cursor object, and it's not obvious what exactly should be passed into the user's sensor function when just a single one of the many asset dependencies has been updated), but I do think that people will look for this functionality pretty quickly, and it would be nice to have a good story for them..

Jun 16 2021, 7:57 PM
owen updated the diff for D8392: Refactor dagster-dbt integration / add dbt cli resource.

rebase

Jun 16 2021, 5:48 PM
owen published D8392: Refactor dagster-dbt integration / add dbt cli resource for review.
Jun 16 2021, 5:08 PM

Jun 10 2021

owen added inline comments to D8330: [dagster-graphql-client] Make `use_https` required.
Jun 10 2021, 4:52 PM

Jun 9 2021

owen closed D8322: Use deepcopy instead of copy in schedule example.
Jun 9 2021, 11:23 PM
owen committed R1:9291a5878174: Use deepcopy instead of copy in schedule example (authored by owen).
Use deepcopy instead of copy in schedule example
Jun 9 2021, 11:23 PM
owen requested review of D8322: Use deepcopy instead of copy in schedule example.
Jun 9 2021, 10:27 PM
owen closed D8278: Hackernews demo in public repo.
Jun 9 2021, 9:06 PM
owen committed R1:c4f1ec6519c9: Hackernews demo in public repo (authored by owen).
Hackernews demo in public repo
Jun 9 2021, 9:06 PM
owen accepted D8313: [dagster-graphql-client] Fixing Exception messages.

nice

Jun 9 2021, 8:47 PM
owen updated the diff for D8278: Hackernews demo in public repo.
  • changed ParquetPointer to NamedTuple
Jun 9 2021, 7:18 PM
owen updated the diff for D8278: Hackernews demo in public repo.
  • added some explanations
Jun 9 2021, 5:44 PM
owen updated subscribers of D8278: Hackernews demo in public repo.

Nice!

After this goes in, what will the procedure be for making changes to the pipelines? Will there be some intermediate period during which we need to replicate changes across both the public repo and the demo repo? Not the end of the world, but ultimately will be pretty painful. Do we have a plan for making the demo depend on the public repo version of the pipeline?

Jun 9 2021, 5:39 PM
owen updated the diff for D8278: Hackernews demo in public repo.
  • added some explanations
Jun 9 2021, 5:28 PM
owen requested review of D8278: Hackernews demo in public repo.
Jun 9 2021, 12:07 AM

Jun 8 2021

owen closed D8225: Move solid error boundary closer to user code invocation to clean up stack traces.
Jun 8 2021, 9:30 PM
owen committed R1:a36e8585045d: Move solid error boundary closer to user code invocation to clean up stack… (authored by owen).
Move solid error boundary closer to user code invocation to clean up stack…
Jun 8 2021, 9:30 PM
owen closed D8262: Create interface for determining python type of all DagsterTypes.
Jun 8 2021, 9:29 PM
owen committed R1:9346729980b0: Create interface for determining python type of all DagsterTypes (authored by owen).
Create interface for determining python type of all DagsterTypes
Jun 8 2021, 9:29 PM
owen updated the diff for D8262: Create interface for determining python type of all DagsterTypes.

up

Jun 8 2021, 9:29 PM
owen added inline comments to D8275: Make solid invocation directly call the underlying decorated fxn.
Jun 8 2021, 9:13 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

rebase

Jun 8 2021, 8:55 PM
owen updated the diff for D8262: Create interface for determining python type of all DagsterTypes.

up

Jun 8 2021, 8:51 PM
owen added inline comments to D8262: Create interface for determining python type of all DagsterTypes.
Jun 8 2021, 6:12 PM
owen updated the diff for D8262: Create interface for determining python type of all DagsterTypes.

up

Jun 8 2021, 5:45 PM
owen added a comment to D8259: [guide] file processing.

Overall, I like the guide, but I think we can simplify it a little bit.

Jun 8 2021, 5:17 PM
owen accepted D8245: expose retry attempts on SoildExecutionContext.
Jun 8 2021, 4:50 PM

Jun 7 2021

owen updated the summary of D8262: Create interface for determining python type of all DagsterTypes.
Jun 7 2021, 7:01 PM
owen requested review of D8262: Create interface for determining python type of all DagsterTypes.
Jun 7 2021, 6:59 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

added some tests for weird things people might do with generators and solids

Jun 7 2021, 5:12 PM
owen requested changes to D8245: expose retry attempts on SoildExecutionContext.
Jun 7 2021, 4:32 PM

Jun 4 2021

owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

up

Jun 4 2021, 10:18 PM
owen accepted D8257: EnvironmentConfig -> ResolvedRunConfig.

nice 👍

Jun 4 2021, 9:28 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

up

Jun 4 2021, 9:21 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

up

Jun 4 2021, 9:20 PM
owen accepted D8256: environment -> run_config cleanups.

I always did wonder why it was called environment in this context 🤔

Jun 4 2021, 8:58 PM
owen accepted D8250: mypy ModeDefinition.

Macro charlie_thumbsup:

Jun 4 2021, 8:25 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

up

Jun 4 2021, 8:24 PM
owen added inline comments to D8225: Move solid error boundary closer to user code invocation to clean up stack traces.
Jun 4 2021, 5:43 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

up

Jun 4 2021, 5:35 PM
owen updated the diff for D8225: Move solid error boundary closer to user code invocation to clean up stack traces.

up

Jun 4 2021, 5:28 PM