Page MenuHomeElementl

Use the EcsRunLauncher in the reference deployment
ClosedPublic

Authored by jordansanders on Jun 23 2021, 2:56 PM.

Details

Summary

Unfortunately, the EcsRunLauncher doesn't support the local development
instructions becuase the local Docker containers can't use

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html

to introspect information about itself. Additionally, we can't use the
AWS SDK to launch local docker containers via the ECS API.

This shouldn't be merged until https://dagster.phacility.com/D8404 lands
and is published to pypi - prior to that, containers will fail to stand up
because the EcsRunLauncher class won't exist yet.

Test Plan

manually tested

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

examples/deploy_ecs/README.md
47

Do we have a pattern for varying our run launcher depending on the environment? Sort of like "modes" but for run launchers? Ideally, we'd use the DefaultRunLauncher when deploying locally and the EcsRunLauncher when deploying remotely.

Fixup the multi-stage target name and give daemon permission to interact with ECS. This IAM policy is a bit unrestricted at the moment - we should scope it down to just its minimum requirements as this matures.

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 23 2021, 3:39 PM
Harbormaster failed remote builds in B32509: Diff 40044!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 23 2021, 5:52 PM
Harbormaster failed remote builds in B32525: Diff 40068!
dgibson added inline comments.
examples/deploy_ecs/Dockerfile
1–18

it would be nice to have some way of knowing before things are pushed to pypi that this example is going to break - deploy_docker has a 'from_source' folder with tests that attempt to provide that guarantee

This revision is now accepted and ready to land.Jun 24 2021, 2:18 PM
examples/deploy_ecs/Dockerfile
1–18

Yeah - getting this under an end-to-end integration test is something I plan to do.