Page MenuHomePhabricator
Differential D756

Add Dagster bash solid
ClosedPublic
Actions

Authored by nate on Aug 2 2019, 6:23 PM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Commits
R1:72062de748f6: Add Dagster bash solid
Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

nate created this revision.Aug 2 2019, 6:23 PM
nate updated this revision to Diff 3377.Aug 2 2019, 6:33 PM

up

Harbormaster failed remote builds in B2690: Diff 3375!Aug 2 2019, 6:36 PM
nate updated this revision to Diff 3378.Aug 2 2019, 6:39 PM

up

Harbormaster completed remote builds in B2691: Diff 3377.Aug 2 2019, 6:42 PM
nate added a reviewer: Restricted Project.Aug 2 2019, 6:43 PM
Harbormaster completed remote builds in B2692: Diff 3378.Aug 2 2019, 6:47 PM
alangenfeld requested changes to this revision.Aug 2 2019, 11:52 PM
alangenfeld added a subscriber: alangenfeld.

hm maybe '[bash][{name}]' instead of '[bash solid]'?

python_modules/libraries/dagster-bash/dagster_bash/solids.py
69–73

ehh this seems like it would suck - maybe we should have an enum to control the output logging behavior BUFFER / STREAM / NONE

This revision now requires changes to proceed.Aug 2 2019, 11:52 PM
alangenfeld added inline comments.Aug 2 2019, 11:53 PM
python_modules/libraries/dagster-bash/dagster_bash/solids.py
69–73

should probably do the same thing for spark too - that has the same bad logging output experience right

nate added inline comments.Aug 3 2019, 12:45 AM
python_modules/libraries/dagster-bash/dagster_bash/solids.py
69–73

yeah that's a good idea - let me at least do it here, I need to think a little more about how best to handle the Spark case

nate updated this revision to Diff 3384.Aug 3 2019, 1:11 AM

comments

Harbormaster completed remote builds in B2698: Diff 3384.Aug 3 2019, 1:25 AM
alangenfeld accepted this revision.Aug 5 2019, 3:59 PM
alangenfeld added inline comments.
python_modules/libraries/dagster-bash/dagster_bash/solids.py
89

maybe output_logging or something else a little more specific, output_type feels too vague

This revision is now accepted and ready to land.Aug 5 2019, 3:59 PM
nate updated this revision to Diff 3401.Aug 5 2019, 5:13 PM

comments

Harbormaster completed remote builds in B2714: Diff 3401.Aug 5 2019, 5:22 PM
Closed by commit R1:72062de748f6: Add Dagster bash solid (authored by nate). · Explain WhyAug 5 2019, 5:24 PM
This revision was automatically updated to reflect the committed changes.
Nate Kupp <nate@elementl.com> added a commit: R1:72062de748f6: Add Dagster bash solid.

Revision Contents
Changeset List

PathSize
python_modules/
dagster/
dagster/
utils/
2 lines
libraries/
dagster-bash/
2 lines
4 lines
dagster_bash/
4 lines
127 lines
3 lines
dagster_bash_tests/
1 line
50 lines
5 lines
51 lines
15 lines

Diff 3403

View Options

python_modules/dagster/dagster/utils/test.py

Loading...
View Options

python_modules/libraries/dagster-bash/.coveragerc

Loading...
View Options

python_modules/libraries/dagster-bash/README.md

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash/__init__.py

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash/solids.py

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash/version.py

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash_tests/__init__.py

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash_tests/test.sh

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash_tests/test_solids.py

Loading...
View Options

python_modules/libraries/dagster-bash/dagster_bash_tests/test_version.py

Loading...
View Options

python_modules/libraries/dagster-bash/requirements.txt

Loading...
View Options

python_modules/libraries/dagster-bash/setup.py

Loading...
View Options

python_modules/libraries/dagster-bash/tox.ini

Loading...