Page MenuHomePhabricator
Feed All Stories

Today

sashank created D1875: Remove status argument from methods.
Thu, Jan 23, 10:20 PM
Harbormaster failed remote builds in B7197: Diff 8870 for D1863: [engine] shared step iteration!
Thu, Jan 23, 9:43 PM
themissinghlink added a comment to D1874: Bay Bikes Weather Pipeline refactor.

ready for a once over! Good call on the naming change and the container switch, this actually makes my general life a lot easier as well since I switched my dagster-home to point to the test-postgres-db.

Thu, Jan 23, 9:31 PM
schrockn added a comment to D1863: [engine] shared step iteration.

"risky" might be the wrong word but more like "too easy to break" meaning that anytime you change that config key it will certainly not behave the way you want to. Any change with the config would have to be coupled with a change to code. Given that it doesn't seem like something that should be in config

Thu, Jan 23, 9:21 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

Thu, Jan 23, 9:18 PM
themissinghlink updated the diff for D1874: Bay Bikes Weather Pipeline refactor.
  • renamed dagster_components to common and switched to using the test-postgres-db container
Thu, Jan 23, 9:17 PM
sashank committed R1:8b0966bce055: Fix example schedules (authored by sashank).
Fix example schedules
Thu, Jan 23, 9:17 PM
Sashank Thupukari <s@thupukari.com> closed D1865: Fix example schedules.
Thu, Jan 23, 9:17 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

So the crux of it is:

  • we do dagster/priority and it works for the core executors but (probably?) none of the others which leads to an unfortunate expectation mis-match in a not clearly communicated way (keys just ignored)
  • we do separate keys for each executor to align expectations ie dagster/in-process/priority, dagster-in-process/priority, whatever but thats aesthetically unfortunate and leads to having to set all keys to support different execution modes
Thu, Jan 23, 9:15 PM
schrockn added a comment to D1863: [engine] shared step iteration.

priority?

Thu, Jan 23, 9:09 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

what should the predefined keys be for the in process and multi process engines? I wasn't happy with what I came up with which motivated me to punt it out to config

Thu, Jan 23, 9:09 PM
max committed R1:4b2dbb31fe86: Working K8sRunLauncher (authored by max).
Working K8sRunLauncher
Thu, Jan 23, 8:46 PM
Max Gasner <max.gasner@gmail.com> closed D1873: Working K8sRunLauncher.
Thu, Jan 23, 8:46 PM
alangenfeld planned changes to D1853: handle differing arg and input def ordering.
Thu, Jan 23, 8:39 PM
Harbormaster failed remote builds in B7198: Diff 8871 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 8:37 PM
schrockn requested changes to D1863: [engine] shared step iteration.

So what I was expecting to see here is not a configurable priority key in terms of *configuring* executor on a per-run basis, but instead just having it be part of the executor definition, plus a comparator functions between priorities. Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

Thu, Jan 23, 8:35 PM
Harbormaster failed remote builds in B7199: Diff 8872 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 8:32 PM
max updated the diff for D1873: Working K8sRunLauncher.

Rebase

Thu, Jan 23, 8:26 PM
max updated the diff for D1873: Working K8sRunLauncher.

Nits

Thu, Jan 23, 8:23 PM
max updated the diff for D1873: Working K8sRunLauncher.

blk

Thu, Jan 23, 8:22 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

move sorting out to individual engines. use config to drive what key to use for prioirty on default engines

Thu, Jan 23, 8:18 PM
Harbormaster failed remote builds in B7196: Diff 8869 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 8:05 PM
max updated the diff for D1873: Working K8sRunLauncher.

py27

Thu, Jan 23, 7:51 PM
Harbormaster failed remote builds in B7195: Diff 8868 for D1867: [RFC] Remove schedule id!
Thu, Jan 23, 7:47 PM
Harbormaster failed remote builds in B7194: Diff 8867 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 7:40 PM
Harbormaster failed remote builds in B7193: Diff 8866 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 7:37 PM
sashank updated the diff for D1867: [RFC] Remove schedule id.

rebase

Thu, Jan 23, 7:37 PM
prha added a comment to D1867: [RFC] Remove schedule id.

as in, not having the migration seems fine, when you can accomplish the same thing by running wipe and up

Thu, Jan 23, 7:28 PM
prha requested changes to D1867: [RFC] Remove schedule id.

I think it should be fine, but we should add something to the changelog, saying "Need to run dagster schedule ..."

Thu, Jan 23, 7:27 PM
max updated the diff for D1873: Working K8sRunLauncher.

Fixup py27

Thu, Jan 23, 7:26 PM
max updated the diff for D1873: Working K8sRunLauncher.

Up

Thu, Jan 23, 7:21 PM
max added inline comments to D1873: Working K8sRunLauncher.
Thu, Jan 23, 7:18 PM
sashank updated the diff for D1865: Fix example schedules.

Up

Thu, Jan 23, 7:13 PM
Harbormaster failed remote builds in B7191: Diff 8864 for D1865: Fix example schedules!
Thu, Jan 23, 7:02 PM
sashank updated the diff for D1865: Fix example schedules.

up

Thu, Jan 23, 6:53 PM
sashank committed R1:5b5055468c13: Move dagster_graphql_test context utils to dagster_graphql (authored by sashank).
Move dagster_graphql_test context utils to dagster_graphql
Thu, Jan 23, 6:51 PM
Sashank Thupukari <s@thupukari.com> closed D1866: Move dagster_graphql_test context utils to dagster_graphql.
Thu, Jan 23, 6:51 PM
Harbormaster failed remote builds in B7189: Diff 8860 for D1865: Fix example schedules!
Thu, Jan 23, 6:33 PM
prha added a comment to D1874: Bay Bikes Weather Pipeline refactor.

partition stuff looks good.

Thu, Jan 23, 6:31 PM
Harbormaster failed remote builds in B7188: Diff 8859 for D1866: Move dagster_graphql_test context utils to dagster_graphql!
Thu, Jan 23, 6:31 PM
Harbormaster failed remote builds in B7187: Diff 8858 for D1866: Move dagster_graphql_test context utils to dagster_graphql!
Thu, Jan 23, 6:28 PM
sashank updated the diff for D1866: Move dagster_graphql_test context utils to dagster_graphql.

Fix merge conflict

Thu, Jan 23, 6:27 PM
sashank added a comment to D1867: [RFC] Remove schedule id.

This shouldn't break existing running schedules because the scheduler manages those all by name, correct?

Thu, Jan 23, 6:23 PM
sashank updated the diff for D1865: Fix example schedules.

Add comment about why we need to do scheduler_handle.up

Thu, Jan 23, 6:21 PM
prha added a comment to D1867: [RFC] Remove schedule id.

byebye

Thu, Jan 23, 6:21 PM
prha accepted D1867: [RFC] Remove schedule id.

This shouldn't break existing running schedules because the scheduler manages those all by name, correct?

Thu, Jan 23, 6:20 PM
sashank updated the diff for D1866: Move dagster_graphql_test context utils to dagster_graphql.

Rebase again

Thu, Jan 23, 6:19 PM
sashank updated the diff for D1866: Move dagster_graphql_test context utils to dagster_graphql.

Rebase on master

Thu, Jan 23, 6:16 PM
prha committed R1:1bd13f63cebe: Combine start/launch buttons into a toggleable execute button group (authored by prha).
Combine start/launch buttons into a toggleable execute button group
Thu, Jan 23, 6:16 PM
prha closed D1871: Combine start/launch buttons into a toggleable execute button group.
Thu, Jan 23, 6:15 PM
alangenfeld resigned from D1874: Bay Bikes Weather Pipeline refactor.
Thu, Jan 23, 6:04 PM
alangenfeld committed R1:ddcfd23617b6: Fix issue with pyspark_solid decorator not handling inputs (authored by nate).
Fix issue with pyspark_solid decorator not handling inputs
Thu, Jan 23, 5:39 PM
alangenfeld closed D1765: Fix issue with pyspark_solid decorator not handling inputs.
Thu, Jan 23, 5:38 PM
alangenfeld accepted D1873: Working K8sRunLauncher.
Thu, Jan 23, 5:38 PM
alangenfeld added a comment to D1873: Working K8sRunLauncher.

changes here all seem reasonable to me

Thu, Jan 23, 5:38 PM
alangenfeld added inline comments to D1873: Working K8sRunLauncher.
Thu, Jan 23, 5:38 PM
alangenfeld accepted D1871: Combine start/launch buttons into a toggleable execute button group.

I think this is better

Thu, Jan 23, 5:22 PM
alangenfeld updated the diff for D1765: Fix issue with pyspark_solid decorator not handling inputs.

rebase

Thu, Jan 23, 5:19 PM
alangenfeld added a comment to D1864: RFC: is_optional --> is_required.

dont have strong feelings - but the complexity cost of warn here feels pretty minimal so i maybe lean that way

Thu, Jan 23, 5:10 PM
schrockn added a comment to D1864: RFC: is_optional --> is_required.

cool. it's not too bad. where do you land on breaking vs warn?

Thu, Jan 23, 4:59 PM
alangenfeld added a comment to D1864: RFC: is_optional --> is_required.

can you explain automated comment. i think i am missing something

Thu, Jan 23, 4:54 PM
schrockn added a comment to D1855: Gaant-style execution plan visualization behind feature flag.

A couple pieces of feedback you can address here or in a follow-on:

Thu, Jan 23, 1:56 PM
schrockn abandoned D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

Abandoning for now.

Thu, Jan 23, 1:28 PM
schrockn added a comment to D1860: Basic composite expansion in the Dagit explore tab.

How difficult would it be to make this not required a full reload? E.g. have the flatten composites button/check stay in view while only dag view reloads. Extra credit for some sort of animation or fade.

Thu, Jan 23, 1:22 PM
schrockn abandoned D1751: configurable_dict_type.
Thu, Jan 23, 1:18 PM
schrockn abandoned D1748: Checking out D1738.
Thu, Jan 23, 1:17 PM
schrockn abandoned D1747: Use integration images for airflow base.
Thu, Jan 23, 1:17 PM
schrockn abandoned D1736: remove type check within type check in define_python_dagster_type.
Thu, Jan 23, 1:17 PM
themissinghlink added reviewers for D1874: Bay Bikes Weather Pipeline refactor: prha, sashank, alangenfeld.
Thu, Jan 23, 6:43 AM
themissinghlink updated the diff for D1874: Bay Bikes Weather Pipeline refactor.
  • tweaked test because c/i has no api key installed
Thu, Jan 23, 6:38 AM
Harbormaster failed remote builds in B7184: Diff 8853 for D1874: Bay Bikes Weather Pipeline refactor!
Thu, Jan 23, 6:36 AM
themissinghlink removed reviewers for D1874: Bay Bikes Weather Pipeline refactor: prha, alangenfeld, sashank.
Thu, Jan 23, 6:35 AM
themissinghlink updated the summary of D1874: Bay Bikes Weather Pipeline refactor.
Thu, Jan 23, 6:35 AM
themissinghlink created D1874: Bay Bikes Weather Pipeline refactor.
Thu, Jan 23, 6:27 AM
Harbormaster failed remote builds in B7183: Diff 8852 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 2:42 AM
max created D1873: Working K8sRunLauncher.
Thu, Jan 23, 2:33 AM
Harbormaster failed remote builds in B7182: Diff 8850 for D1867: [RFC] Remove schedule id!
Thu, Jan 23, 1:55 AM
prha committed R1:4b1842251958: add collapsed tags view in RunTable (authored by prha).
add collapsed tags view in RunTable
Thu, Jan 23, 1:52 AM
prha closed D1861: add collapsed tags view in RunTable.
Thu, Jan 23, 1:52 AM
sashank updated the diff for D1867: [RFC] Remove schedule id.

rebase on master

Thu, Jan 23, 1:46 AM
prha added reviewers for D1871: Combine start/launch buttons into a toggleable execute button group: bengotow, alangenfeld.
Thu, Jan 23, 1:36 AM
prha updated the summary of D1871: Combine start/launch buttons into a toggleable execute button group.
Thu, Jan 23, 1:35 AM
prha updated the diff for D1871: Combine start/launch buttons into a toggleable execute button group.

handle small button sizes correctly

Thu, Jan 23, 1:35 AM
prha updated the diff for D1871: Combine start/launch buttons into a toggleable execute button group.

rebase

Thu, Jan 23, 1:20 AM
Harbormaster failed remote builds in B7179: Diff 8847 for D1871: Combine start/launch buttons into a toggleable execute button group!
Thu, Jan 23, 1:17 AM
themissinghlink abandoned D1833: Add schedules to weather pipelines.

Realized that if we wanted to switch to the partition API, we need to design my pipelines to require transactional consistency. cloud storage is not that, so we need to switch to using a database. I should have done this a long time ago, but am going to do a bunch of incremental refactors to get there instead of blowing up this diff.

Thu, Jan 23, 1:11 AM
prha updated the diff for D1871: Combine start/launch buttons into a toggleable execute button group.

prevent local storage clobber

Thu, Jan 23, 1:06 AM
Harbormaster failed remote builds in B7178: Diff 8845 for D1864: RFC: is_optional --> is_required!
Thu, Jan 23, 12:45 AM
themissinghlink committed R1:81745d294c4c: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores (authored by themissinghlink).
BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores
Thu, Jan 23, 12:42 AM
themissinghlink closed D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.
Thu, Jan 23, 12:42 AM
Harbormaster failed remote builds in B7176: Diff 8843 for D1867: [RFC] Remove schedule id!
Thu, Jan 23, 12:37 AM
schrockn updated the diff for D1864: RFC: is_optional --> is_required.

rebase

Thu, Jan 23, 12:36 AM
themissinghlink updated the diff for D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.

Repushing to bring in buildkite nbconvert issues.

Thu, Jan 23, 12:31 AM
sashank updated the diff for D1867: [RFC] Remove schedule id.

Fix AppCache dataIdFromObject for RunningSchedule

Thu, Jan 23, 12:25 AM
schrockn committed R1:75d791424874: Only do python3 support for pipenv smoke (authored by schrockn).
Only do python3 support for pipenv smoke
Thu, Jan 23, 12:25 AM
schrockn closed D1872: Only do python3 support for pipenv smoke.
Thu, Jan 23, 12:25 AM
themissinghlink accepted D1872: Only do python3 support for pipenv smoke.

LGTM

Thu, Jan 23, 12:23 AM
Harbormaster failed remote builds in B7173: Diff 8839 for D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores!
Thu, Jan 23, 12:20 AM
schrockn updated the diff for D1872: Only do python3 support for pipenv smoke.

upmessage

Thu, Jan 23, 12:17 AM
schrockn abandoned D1870: test pipenv.
Thu, Jan 23, 12:16 AM