ECS continues to run into list index errors:
I haven't been able to reproduce the issue but my best guess is that we're
running into eventual consistency issues with ECS. This is consistent with these
AWS suggests an expontential backoff of up to 5 minutes. I think that's a little
extreme for our use case - particularly because we don't want to block the
GraphQL query from resolving.
Instead, I'm changing the behavior of .can_terminate to return False if we
run into this eventual consistency. This means occassionally, truly cancellable
pipelines will show as unable to cancel. Fortunately, the value of
.can_terminate isn't memoized so it won't be stuck as uncancellable for the
entire lifetime of the pipeline run.