These pytest fixtures aim to replace much of what we currently do in
https://github.com/dagster-io/dagster/blob/c7cdcbc39d16f81287620f15edf42bc29eec5af0/.buildkite/dagster-buildkite/dagster_buildkite/utils.py#L71-L96
and https://github.com/dagster-io/dagster/blob/master/.buildkite/hooks/pre-exit
The two biggest benefits of this approach are:
- Conditional environment logic (i.e. different behavior when running on Buildkite) is relegated to the fixtures instead of living in the tests. While we'll still have two paths through every test that uses docker compose (one that uses localhost and one that jumps through some separate networking hoops), that branching will only live in a single place instead of repeating it over and over in each test.
- We can build more interesting test scenarios in pure Python. For example, with the old approach, if you wanted to do any setup before docker compose up, you needed to offload that test setup into a script and call it from the Buildkite step. Now, you can handle that test setup in pure Python inside your test fixtures.
There are two types of fixtures being introduced here:
- A higher level docker_compose fixture that's not customizable, but operates like a true Pytest fixture. You can use this if your docker-compose.yml lives in your test directory and if there's no additional setup or customization you need by simply including the fixture name as an argument to your test.
- A lower level docker_compose_cm fixture that yields a customizable contextmanager. This can be chained with other fixtures to produce highly customizable test setup.
These fixtures can be imported individually. Alternatively, as long as
dagster_test is installed, you can have test suites automatically
discover them via any of Pytest's plugin discovery options:
https://docs.pytest.org/en/6.2.x/writing_plugins.html#plugin-discovery-order-at-tool-startup
Initially, I'm setting the pytest_plugins variable in test suites that
use this. Eventually, I'd like to configure Buildkite and our
pyproject.toml to automatically include them in all of our tests - but
doing that requires some more intentional thought around _where_ these
fixtures should live. I see a few different options:
- dagster_test - where I've chosen to place them initially. But I'm not sure we'd ever want this to be a dependency of _all_ libraries.
- dagster - similar to where we put dagster.core.test_utils.py. But it seems odd to include pytest as a dependency of dagster.
- A new library - perhaps dagster-pytest - that gets published to pypi. Although this is all specific to _our_ testing setup on Buildkite. A package with that name might be better as a customer-facing package with fixtures that allow you to easily test your own dagster code.