Page MenuHomePhabricator

[RFC] Parallelize long running buildkite steps
AbandonedPublic

Authored by sashank on Apr 7 2020, 11:05 PM.

Details

Reviewers
max
nate
Summary

Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
timing-tests
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

sashank created this revision.Apr 7 2020, 11:05 PM
sashank updated this revision to Diff 11543.Apr 8 2020, 12:07 AM

double quotes

sashank updated this revision to Diff 11544.Apr 8 2020, 12:20 AM

tox env syntax

sashank updated this revision to Diff 11545.Apr 8 2020, 12:22 AM

turn down parallelism

Harbormaster failed remote builds in B9337: Diff 11545!
sashank retitled this revision from Timing to Parallelize long running buildkite steps.
sashank edited the summary of this revision. (Show Details)
sashank added reviewers: max, nate.
sashank added inline comments.Apr 8 2020, 1:50 AM
examples/airline.tox
46

This won't work locally rn - need to update to not use the split groups when these env vars are not defined

python_modules/dagster/dev-requirements.txt
16

need to pin this

sashank retitled this revision from Parallelize long running buildkite steps to [RFC] Parallelize long running buildkite steps.Apr 8 2020, 1:51 AM
sashank updated this revision to Diff 11557.Apr 8 2020, 3:09 AM

tweaking some numbers

sashank planned changes to this revision.Apr 8 2020, 3:15 AM

Hm the speedup isn't that significant when we run all the tests, but it does seem like at least 1 - 2 minute speedup. Doesn't feel like it's worth it. Will play around with this more

sashank abandoned this revision.Fri, May 15, 5:02 AM