I realize this is totally out of scope for this diff, but this is my first time seeing the python syntax for this. I was sort of envisioning these decorators would be added to standard steps and we wouldn't have a formalized concept of an asset step vs a freeform step? I'm a bit worried that in this world, if I have one step that doesn't cleanly behave like an asset (maybe just some basic loader step that gets me USD exchange rates i use downstream in my pipeline), I can't use the asset concepts in my job? It seems like people will end up doing a ton of work inside single asset transforms rather than breaking things out into a nice graph of re-usable pieces?

@bengotow I appreciate you weighing in on this stuff!

One way that I think about the issue you raised is as a friction between two models of specifying dependencies:

• In the Airflow/Prefect/current Dagster model of specifying dependencies, you start with an independent set of steps/solids and chain them together into a DAG.
• In the software-defined assets model, dependencies are part of the identity of an asset.

IMO, including the dependencies as part of the identity of an asset is a crucial piece of the approach. Here are a few of the pieces why:

• Most data derivation functions only make sense in the context of the data they run on. E.g. a pretty typical node in a graph would be something like "build a clean version of the messy user_events table". It doesn't make sense to even really talk about that op without talking about the user_events table it depends on. Kind of like how, in React, you directly reference subcomponents when defining a component. You wouldn't say "this is a generic function that can take any sub-component and surround it in a border" and then string the component hierarchy together elsewhere. (Or maybe I'm making false claims about React, I'm not a big user, plz call me out if so). The component (aka asset) is reusable, but the function that builds it from its sub-components (aka dependencies) is less so.
• If dependencies are determined when defining the graph, then, if you have 400 assets, you end up needing to have a single function that encapsulates all of the dependencies between those assets. Composition helps a little bit, but it still requires a sort of Stalinist centrally-planned regime.
• When you build a dataset, e.g. in a Jupyter notebook, you put the transformations and dependencies in one place.

When you say "cleanly behave like an asset", are there particular violations you have in mind? The basic loader step that you described above sounds like a good fit for an asset.

Happy to talk more about this.

Hey Sandy! Thanks for the detailed explanation I think that clarifies things a lot. I'm doing more mockup work today and I think I'm on track.

I've been thinking of "asset" as "meaningful work product persisted to S3, etc.", and I think that might be incorrect? It sounds like in this model you'd have a job with a dependency graph of assets, and each transformed intermediate is an "asset" even if the only business product you care about is the output of the last step?

That's right - we'll probably need some way of designating assets as important vs. intermediate so that people viewing the asset graph can filter down.

That said, I think it's fairly common for every step in a pipeline to produce an asset that's meaningful externally. For example, the outputs of the solids in the story recommender demo pipeline are:

• A table that links comments to the Hacker News stories they sit under.
• A matrix that records how many times each user has commented on each story.
• A recommendation model.
• A table of recommendations.
• A table that helps describe the recommendation model.

If I were building this in real life, I would consider all of these "meaningful work products" and persist them in storage where others could access them.

by making this repository-scoped, we're deliberately ignoring any multi-repo case, correct?