Page MenuHomeElementl

Terminate ECS runs
ClosedPublic

Authored by jordansanders on Jun 16 2021, 4:07 PM.

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

jordansanders created this revision.

Remove unused ecs fixture argument

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2021, 4:54 PM
Harbormaster failed remote builds in B32173: Diff 39634!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2021, 6:11 PM
Harbormaster failed remote builds in B32203: Diff 39669!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2021, 7:54 PM
Harbormaster failed remote builds in B32221: Diff 39689!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2021, 8:29 PM
Harbormaster failed remote builds in B32239: Diff 39707!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 17 2021, 3:53 PM
Harbormaster failed remote builds in B32289: Diff 39767!
dgibson added inline comments.
python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/test_launcher.py
87–94

this is all still just stubbed right? termination is tricky to get right, would be nice to get some coverage that sees if it works in a real ECS environment. Did you test manually terminating against a real ECS task?

This revision is now accepted and ready to land.Jun 17 2021, 10:25 PM
python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/test_launcher.py
87–94

Correct. What I plan to do is use the stubs for actual development/unit testing with a faster iteration loop but then add a higher level end-to-end test that runs against real ECS. It'll docker-compose up a daemon and a user code repository onto ECS (which will scaffold all the ECS resources with tags), run a pipeline, and check that it worked. We can also run a longer-running pipeline and test termination. And then it'll docker-compose down.

I don't want to start with this approach though for a few reasons (that all more or less get summed up as "garbage collection"):

  • if docker-compose up stands up an ECS task that can't actually run (like invalid config or something), we get stuck in an infinite loop where the CloudFormation stack never finishes applying. So we can never use docker compose down to tear it down (I believe the solution for this is to enable deployment circuit breaking https://aws.amazon.com/blogs/containers/announcing-amazon-ecs-deployment-circuit-breaker/)
  • docker-compose up adds tags to all the resources it creates so I want to write a cleanup method that finds everything with those tags and destroys them (in case docker-compose down fails or is just incomplete)
This revision was automatically updated to reflect the committed changes.