Page MenuHomeElementl
Differential D8492

Use the EcsRunLauncher in the reference deployment
ClosedPublic
Actions

Authored by • jordansanders on Jun 23 2021, 2:56 PM.
Tags
None
Referenced Files
F2982867: D8492.diff
Sun, May 28, 9:56 AM
Unknown Object (File)
Sat, May 27, 12:51 AM
Unknown Object (File)
Tue, May 23, 5:30 AM
Unknown Object (File)
Fri, May 19, 8:39 PM
Unknown Object (File)
Tue, May 16, 7:47 PM
Unknown Object (File)
Mon, May 15, 9:05 PM
Unknown Object (File)
Sun, May 14, 12:01 PM
Unknown Object (File)
Tue, May 9, 5:36 PM
View All Files
Subscribers
None

Details

Reviewers
• dgibson
• johann
• catherinewu
Commits
R1:f331e0eafafe: Use the EcsRunLauncher in the reference deployment
Summary

Unfortunately, the EcsRunLauncher doesn't support the local development
instructions becuase the local Docker containers can't use

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html

to introspect information about itself. Additionally, we can't use the
AWS SDK to launch local docker containers via the ECS API.

This shouldn't be merged until https://dagster.phacility.com/D8404 lands
and is published to pypi - prior to that, containers will fail to stand up
because the EcsRunLauncher class won't exist yet.

Test Plan

manually tested

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

• jordansanders created this revision.Jun 23 2021, 2:56 PM
• jordansanders added inline comments.Jun 23 2021, 2:57 PM
examples/deploy_ecs/README.md
47

Do we have a pattern for varying our run launcher depending on the environment? Sort of like "modes" but for run launchers? Ideally, we'd use the DefaultRunLauncher when deploying locally and the EcsRunLauncher when deploying remotely.

• jordansanders added a parent revision: D8491: Cross-reference ECS reference deployment.Jun 23 2021, 2:57 PM
• jordansanders updated this revision to Diff 40037.Jun 23 2021, 3:03 PM

rebase

• jordansanders updated this revision to Diff 40038.Jun 23 2021, 3:07 PM

Fixup the multi-stage target name and give daemon permission to interact with ECS. This IAM policy is a bit unrestricted at the moment - we should scope it down to just its minimum requirements as this matures.

• jordansanders updated this revision to Diff 40044.Jun 23 2021, 3:15 PM

rebase

Harbormaster failed remote builds in B32496: Diff 40031!Jun 23 2021, 3:18 PM
Harbormaster failed remote builds in B32502: Diff 40037!Jun 23 2021, 3:30 PM
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 23 2021, 3:39 PM
Harbormaster failed remote builds in B32509: Diff 40044!
Harbormaster failed remote builds in B32503: Diff 40038!Jun 23 2021, 3:50 PM
• jordansanders updated this revision to Diff 40068.Jun 23 2021, 5:28 PM

rebase

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 23 2021, 5:52 PM
Harbormaster failed remote builds in B32525: Diff 40068!
• jordansanders updated this revision to Diff 40093.Jun 23 2021, 8:46 PM

rebase

• jordansanders edited the summary of this revision. (Show Details)Jun 23 2021, 8:46 PM
Harbormaster completed remote builds in B32543: Diff 40093.Jun 23 2021, 9:12 PM
• jordansanders requested review of this revision.Jun 23 2021, 9:12 PM
• dgibson accepted this revision.Jun 24 2021, 2:18 PM
• dgibson added inline comments.
examples/deploy_ecs/Dockerfile
1–18

it would be nice to have some way of knowing before things are pushed to pypi that this example is going to break - deploy_docker has a 'from_source' folder with tests that attempt to provide that guarantee

This revision is now accepted and ready to land.Jun 24 2021, 2:18 PM
• jordansanders added inline comments.Jun 24 2021, 2:21 PM
examples/deploy_ecs/Dockerfile
1–18

Yeah - getting this under an end-to-end integration test is something I plan to do.

• jordansanders updated this revision to Diff 40129.Jun 24 2021, 2:22 PM

update aws docs

Harbormaster completed remote builds in B32573: Diff 40129.Jun 24 2021, 2:46 PM
Closed by commit R1:f331e0eafafe: Use the EcsRunLauncher in the reference deployment (authored by • jordansanders). · Explain WhyJun 24 2021, 3:37 PM
This revision was automatically updated to reflect the committed changes.
• jordansanders added a commit: R1:f331e0eafafe: Use the EcsRunLauncher in the reference deployment.

Revision Contents
Changeset List

PathSize
docs/
content/
deployment/
guides/
2 lines
examples/
deploy_ecs/
2 lines
20 lines
4 lines
6 lines

Diff 40135

View Options

docs/content/deployment/guides/aws.mdx

Loading...
View Options

examples/deploy_ecs/Dockerfile

Loading...
View Options

examples/deploy_ecs/README.md

Loading...
View Options

examples/deploy_ecs/dagster.yaml

Loading...
View Options

examples/deploy_ecs/docker-compose.yml

Loading...