Testing options for ECS are limited:
- Moto's mocking of ECS is incomplete
- Localstack's ECS implementation is part of their Pro offering and it's not immediately clear that it'd be sufficient
- Integration testing against a live endpoint is slower, requires additional infrastructure management, and require real credentials in Buildkite. Also, many of the ECS operations are immutable. For example, once we register a task definition, we can never delete it - we can only mark it as inactive.
Instead, I've opted to stub the ECS endpoints that we'll need for an
EcsRunLauncher beginning with RegisterTaskDefinition:
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RegisterTaskDefinition.html
To add some additional confidence to the stubs, I'm using botocore's
Stubber:
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/stubber.html
Stubber validates both parameters and responses - it won't allow you to
stub an invalid response structure and it won't allow you to pass
invalid parameters. However, some internal behaviors of ECS still need
to be faked. For example, when registering a task definition to the same
family, ECS automatically increments the revision number. In our
StubbedEcs, this is achieved by storing task definitions in an in memory
dictionary.
While subject to all the usual downsides of stubbing, this approach
gives me enough confidence to meaningfully unit test against ECS. It
doesn't prevent us from eventually adding a handful of end-to-end tests
against a live ECS endpoint.