POC for multiqueue execution in Celery engine.
k let's move forward.
for posterity's sake I think we should figure out a way to locally test this without any sort of docker dependency at all. with multiple processes and a persistence queue somewhere it should be possible. I'm not a celery expert by any means but they certainly claim to have a local development story in their documentation. engine.py should have code coverage without a docker image.
here is coverage in the status quo (running with no queue):
dagster_celery/engine.py 62 2 26 3 94% 83-86, 80->79, 92->91, 121->120 dagster_celery/tasks.py 29 1 4 1 94% 51, 50->51
in my experience, the most dangerous parts of a system like this are never in user code
no, the test setup works fine in a local integration image container