We see pipeline as queued -> dequeuer takes it off queue, launches it -> we mark pipeline as failed -> run worker marks pipeline as NOT_STARTED

The only solution I've thought of so far is to use full transactions between the event and run stores. E.g something like

if run_state = QUEUED
    add_event(RUN_CANCELLED)
else:
    abort

This involves combining event log and run store. (Or we could make run states not tied to event store, but that's a pretty core assumption so far)

Typo, should be

We see pipeline as queued -> dequeuer takes it off queue, places it in NOT_STARTED, launches it -> we mark pipeline as failed -> run worker marks pipeline as STARTED

Now the user might have seen the pipeline as cancelled in dagit, but it's continuing to execute. Dagit will see conflicting events in the log and have potentially unexpected behavior.

yeah this is tricky.One thing that we could do is have cancelling here just set a flag that we then expect the daemon and run worker to periodically check for? The race-y part of this is the fact that the dequeuer overwrites the change that we make here (the run status). If this cancel operation set some other flag that doesn't get overwritten, the race goes away right?

and sorry the implication there would be that this process wouldn't actually do the status update as well - it would be the daemon and/or run worker that do the actual run status update, at the same time that execution is actually halted

instance.report_run_failed