Page MenuHomePhabricator

Break integration tests out into separate Python file
ClosedPublic

Authored by max on Oct 23 2020, 5:38 PM.

Details

Summary

This is prep for triggering integration tests on a separate schedule against master.

Test Plan

N/A

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There's some risk involved in this, but to me it's far outweighed by cutting the build cycle time in half.

max requested review of this revision.Oct 23 2020, 5:54 PM

Great! Adding @catherinewu also.

Per our discussion on Slack, can we have something in place for getting build failures to show up in Slack before landing?

ah, also I think we need to move the test image builds to this pipeline, from pipeline.py - see test_image_depends_fn

certainly need to deal with the test image deps as nate mentions above

If we are going to do this - should we go for broke and move all the tests that depend on test images out to this separate pipeline? Worth looking in to the BK tools for triggering pipelines from other pipelines - we could run the integration tests pipeline automatically on diffs with certain criteria in a similar way we only run js tests on js-only diffs [1]

.buildkite/pipeline.py
781–791

[1]

This revision now requires changes to proceed.Oct 27 2020, 4:44 PM

If we are going to do this - should we go for broke and move all the tests that depend on test images out to this separate pipeline? Worth looking in to the BK tools for triggering pipelines from other pipelines - we could run the integration tests pipeline automatically on diffs with certain criteria in a similar way we only run js tests on js-only diffs [1]

i'd want to do this in a follow-on if we do it. i have been burned hard by overzealous assumptions about what might possibly break from changes in various parts of a codebase before.

See one nit above, otherwise LGTM.

Can you also add a README or some documentation explaining how everyone should run integration tests for their diffs after this lands? And maybe document what schedule this will run on?

Also, can we have this pipeline always run on pushes to master?

.buildkite/integration.py
6

nit, can we move these out to a separate python file, and then have the main pipeline & this pipeline import from there?

This revision is now accepted and ready to land.Mon, Nov 2, 8:53 PM