Adds basic docs for execution using dagster-celery.
Looking nice. Thanks for doing this.
let's omit BigQuery and Dataproc stuff for now. i want to predicate the inclusion of document on an audit of these libraries. I'm not convinced that these are patterns that we want the ecosystem to cargo cult
In production, more configuration is required
may want to note that this is typically at DAGSTER_HOME
I would with why one has to do this.
Because data is passed between solids potentially running on different nodes, you must use storage that is accessible from all the nodes in the celery cluster. Examples include s3 or gcs. An appropriate system_storage -- such as s3_system_storage or gcs_strorage -- must be available in the ModeDefinition
|2–19 ↗||(On Diff #9536)|
levers -> leverages?
|1–19 ↗||(On Diff #9536)|
this feels like scratch work? clean up and handle in other k8s focused diff?
feel like this should be documented as well as its siblings
Just a brain dump of questions I had while reading this
Can we link to: http://docs.celeryproject.org/en/latest/getting-started/brokers/ here to provide instructions on how to use other brokers + setups straight from the source
Why does it need to be from the same directory? It would be good to highlight what implicit dependence there is on the directory I run the command from
I think running it like this wouldn't use the celery executor? If so, we should clarify that this is just a sanity check.
Also, we should explicitly explain how adding the config below enables the celery executor. Also, it might be a good idea to explain why we need `storage: filesystem"
It looks like I need to do this from dagit. Probably should make that explicit, or show the command to execute w/ config from the command line.
Don't really understand what you mean by the code is available to both Dagit and the worker, and why running the worker in the same directory would enable this.