makes sense -- I only kept it around in case we ended up using it for something, but I suppose git history will do.
Aug 17 2021
Aug 9 2021
Aug 4 2021
totally agree with getting rid of this -- it'll also make the dogfooding alert channel much less annoying :)
Aug 2 2021
Like the general shape of this + how it makes the interface feel more similar to @op
Jul 30 2021
There are a couple things I'm worried are missing from solid hooks:
- Re-execution and retries. E.g. imagine you refresh an asset that's a table with a set of emails you want to send, and then you want to send the emails. If the email-sending code fails to connect to the email-sending service, you'd want to be able to re-execute that step without also re-executing the step that generates the table.
- Executing in separate processes. Generating a table might require running a Spark job on a special Spark cluster or doing inference on a GPU. If you want to do a time-consuming operation afterwards (like send a bunch of emails) that doesn't need the cluster or GPU, it would be wasteful to use the same process.
Jul 29 2021
@sandyryza Writing out / formalizing some stuff from our conversation yesterday, it feels like there are two distinct cases where a person might want to include things that don't produce "assets" (long-living objects external to the dagster ecosystem) within their asset oriented pipeline.
Jul 28 2021
Jul 27 2021
Jul 26 2021
One interesting problem that we've discussed in past iterations of this direction is how different @assets may materialize their input assets in to different in-memory types, controlled separately from how they are persisted. Thoughts on that? Something for a follow up?
I think this is a question even in the non-@asset world of solids and IOManagers. Our current recommendation is that, in their IOManager, users switch on the Python type attached to the InputDefinition. Here's an example in the hacker news pipeline: https://github.com/dagster-io/dagster/blob/master/examples/hacker_news/hacker_news/resources/parquet_io_manager.py#L46.
Jul 23 2021
Jul 21 2021
not sure what the deal is with pokemon in reviews but when in rome...
Jul 20 2021
Jul 19 2021
Jul 16 2021
Jul 14 2021
Nice -- one step closer to banishing solids from this world
Jul 13 2021
Jul 12 2021
one other option that might work would be to remove the PROFILES_DIR configuration (https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/dbt_example/dbt_example/solids.py?L11). It seems like the value that it's set to is the default value for dbt, so it doesn't really do anything, and might be having some weird interaction with the update. The changelog for dbt mentions some changes to how they interpret the profiles-dir path (relative to the project-dir path vs absolute) that I wouldn't think would impact us, but it seems possible that this would be a buggy side effect
Jul 6 2021
This approach makes sense to me. But let's make sure this content doesn't conflict with the "recommended pattern" below https://docs.dagster.io/concepts/types#associating-dagster-types-with-python-types
- add extra explanation
- Updated docs for dbt integration
Jul 1 2021
- major refactor
- make black
- major refactor
Jun 25 2021
Jun 16 2021
Overall, I like this direction, but I do think there are many cases where you might want to depend on multiple asset keys at once (the story_recommender pipeline is one example). This does add bits of complexity basically everywhere (need a fancier cursor object, and it's not obvious what exactly should be passed into the user's sensor function when just a single one of the many asset dependencies has been updated), but I do think that people will look for this functionality pretty quickly, and it would be nice to have a good story for them..
Jun 10 2021
Jun 9 2021
- changed ParquetPointer to NamedTuple
- added some explanations
After this goes in, what will the procedure be for making changes to the pipelines? Will there be some intermediate period during which we need to replicate changes across both the public repo and the demo repo? Not the end of the world, but ultimately will be pretty painful. Do we have a plan for making the demo depend on the public repo version of the pipeline?
- added some explanations
Jun 8 2021
Overall, I like the guide, but I think we can simplify it a little bit.
Jun 7 2021
added some tests for weird things people might do with generators and solids
Jun 4 2021
I always did wonder why it was called environment in this context 🤔