Page MenuHomePhabricator

drop legacy_examples dependency outside examples/ #2653
ClosedPublic

Authored by yuhan on Jul 2 2020, 12:21 AM.

Details

Summary

https://github.com/dagster-io/dagster/issues/2653
copy-n-pasted pipeline files used by others from legacy_examples/toys to dagster-test/dagster_test/toys -- the idea is to put toy pipelines used by tests in dagster-test

old dagster-test

dagster_test/
	test_project/
		Dockerfile
		build.sh
		test_pipelines/
			setup.py
			workspace.yaml
			environments/
			test_pipelines/
	test_project.py
dagster_test_tests/

new

dagster_test/
	dagster_airflow/
	toys/
	test_project/
		__init__.py (previously test_project.py)
		test_pipelines/
			workspace.yaml
			environments/
			test_pipelines/
dagster_test_tests/
Dockerfile
build.sh
Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

yuhan created this revision.Jul 2 2020, 12:21 AM
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 12:26 AM
Harbormaster failed remote builds in B14572: Diff 17871!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 12:40 AM
Harbormaster failed remote builds in B14574: Diff 17874!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 12:55 AM
Harbormaster failed remote builds in B14581: Diff 17881!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 4:16 AM
Harbormaster failed remote builds in B14584: Diff 17885!
yuhan updated this revision to Diff 17934.Jul 2 2020, 7:23 PM

test_pipelines is no longer its own module

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 7:32 PM
Harbormaster failed remote builds in B14616: Diff 17934!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 7:45 PM
Harbormaster failed remote builds in B14618: Diff 17936!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 7:59 PM
Harbormaster failed remote builds in B14621: Diff 17940!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 8:41 PM
Harbormaster failed remote builds in B14626: Diff 17948!
Harbormaster returned this revision to the author for changes because remote builds failed.Jul 2 2020, 9:29 PM
Harbormaster failed remote builds in B14632: Diff 17957!
yuhan edited the summary of this revision. (Show Details)Jul 2 2020, 9:46 PM
yuhan requested review of this revision.Jul 2 2020, 9:46 PM
yuhan updated this revision to Diff 17961.Jul 2 2020, 9:48 PM

delete unused file

This is an excellent change.

My only question is what is the long-term vision for dagster-test?

I am going to propose adding a separation section in our codebase for integration tests. Test that require heavy things like the test image, or all the heavy dependencies in the integration image.

I'm fine with dagster-test just being "all the stuff we need for tests" for now, but I was wondering if you had something more concrete in mind?

yuhan added a comment.Mon, Jul 6, 5:54 PM

@schrockn I see dagster-test as a place to put all the code shared by many python modules so we don't need to replicate test snippets or use code in examples/.
integration tests wise, i dont have strong opinions on where to put, but slightly prefer to have it separate from dagster-test and just keep dagster-test a simple "shared examples used by tests"

schrockn accepted this revision.Mon, Jul 6, 6:22 PM
This revision is now accepted and ready to land.Mon, Jul 6, 6:22 PM
yuhan updated this revision to Diff 18064.Mon, Jul 6, 7:10 PM

rebase

This revision was automatically updated to reflect the committed changes.