Give dagit permission to interact with ECS
ClosedPublic
Actions

Authored by • jordansanders on Jul 20 2021, 9:46 PM.

Details

Reviewers

• dgibson
• johann
• max

Commits

R1:538c27bcada0: Give dagit permission to interact with ECS

Summary

Two different users encountered suspiciously similar issues with
ECS depoyments today:

https://github.com/dagster-io/dagster/issues/4377

and

https://dagster.slack.com/archives/C01U954MEER/p1626814414314800

I was able to reproduce the issue by launching a run and then
immediately switching to view in progress runs. I started seeing GraphQL
failures indicating NoCredentialsError. I suspect this is because in our
reference ECS deployment, dagit indeed doesn't have the ability to
interact with the ECS SDK.

This adds a TaskRole to dagit with the same widely-scoped access as the
one we attach to the daemon task. We should more narrowly scope both but
I'll leave that task to another day.

Test Plan

I tested this by redeploying using the updated docker-compose.yml and
verifying that:

a) the TaskRole is attached
b) I can't reproduce the error

Diff Detail

Repository

R1 dagster

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

• jordansanders created this revision.Jul 20 2021, 9:46 PM

• jordansanders mentioned this in D8978: Infer cluster from tags.Jul 20 2021, 9:50 PM

• jordansanders added a child revision: D8978: Infer cluster from tags.Jul 20 2021, 9:50 PM

Harbormaster completed remote builds in B33947: Diff 41934.Jul 20 2021, 10:12 PM

• jordansanders requested review of this revision.Jul 20 2021, 10:12 PM

in addition to this, do we want to make any python changes so that the error message in https://dagster.slack.com/archives/C01U954MEER/p1626814436315000?thread_ts=1626814414.314800&cid=C01U954MEER is less cryptic? a descriptive exception from can_terminate seems better than "list index out of range"

This revision is now accepted and ready to land.Jul 21 2021, 7:34 PM