Maybe rather than having the status checks at all (this diff), we should just implement the k8s can terminate check? Due to the negative effect listed below
Unify can_terminate logic for run_launchers-
Currently the k8s run launchers look at pipeline status, while the others use their specific api. This removes these pipeline status concerns from the run launchers and places them on the instance, so that the launchers purely interact with their specific api. This helps in cases where only the api calls are desired.
- in rare cases, this change may make some runs unable to terminate without force set- e.g. if docker run launcher starts a container but the container doesn't get far enough to emit started, it could currently still be terminated.
- haven't done a thorough review of the other launchers