Like the general shape of this + how it makes the interface feel more similar to @op

In D9074#237104, @sandyryza wrote:

There are a couple things I'm worried are missing from solid hooks:

• Re-execution and retries. E.g. imagine you refresh an asset that's a table with a set of emails you want to send, and then you want to send the emails. If the email-sending code fails to connect to the email-sending service, you'd want to be able to re-execute that step without also re-executing the step that generates the table.
• Executing in separate processes. Generating a table might require running a Spark job on a special Spark cluster or doing inference on a GPU. If you want to do a time-consuming operation afterwards (like send a bunch of emails) that doesn't need the cluster or GPU, it would be wasteful to use the same process.

@sandyryza Writing out / formalizing some stuff from our conversation yesterday, it feels like there are two distinct cases where a person might want to include things that don't produce "assets" (long-living objects external to the dagster ecosystem) within their asset oriented pipeline.

In D8969#234895, @sandyryza wrote:

@alangenfeld

One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?

I think this is a question even in the non-@asset world of solids and IOManagers. Our current recommendation is that, in their IOManager, users switch on the Python type attached to the InputDefinition. Here's an example in the hacker news pipeline: https://github.com/dagster-io/dagster/blob/master/examples/hacker_news/hacker_news/resources/parquet_io_manager.py#L46.

not sure what the deal is with pokemon in reviews but when in rome...

Nice -- one step closer to banishing solids from this world

one other option that might work would be to remove the PROFILES_DIR configuration (https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/dbt_example/dbt_example/solids.py?L11). It seems like the value that it's set to is the default value for dbt, so it doesn't really do anything, and might be having some weird interaction with the update. The changelog for dbt mentions some changes to how they interpret the profiles-dir path (relative to the project-dir path vs absolute) that I wouldn't think would impact us, but it seems possible that this would be a buggy side effect

In D8691#227443, @yuhan wrote:

This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types

• add extra explanation

• rebase

• rebase

• Updated docs for dbt integration

• up

• rebase

• major refactor
• make black

• major refactor

Overall, I like this direction, but I do think there are many cases where you might want to depend on multiple asset keys at once (the story_recommender pipeline is one example). This does add bits of complexity basically everywhere (need a fancier cursor object, and it's not obvious what exactly should be passed into the user's sensor function when just a single one of the many asset dependencies has been updated), but I do think that people will look for this functionality pretty quickly, and it would be nice to have a good story for them..

rebase

nice

• changed ParquetPointer to NamedTuple

• added some explanations

In D8278#217330, @sandyryza wrote:

Nice!

After this goes in, what will the procedure be for making changes to the pipelines? Will there be some intermediate period during which we need to replicate changes across both the public repo and the demo repo? Not the end of the world, but ultimately will be pretty painful. Do we have a plan for making the demo depend on the public repo version of the pipeline?

• added some explanations

up

rebase

up

up

Overall, I like the guide, but I think we can simplify it a little bit.

added some tests for weird things people might do with generators and solids

up

nice 👍

up

up

I always did wonder why it was called environment in this context 🤔

up

up

up

up