Page MenuHomePhabricator

Open Tasks

  • No tasks found.

Active Repositories

Recent Activity

Today

Harbormaster failed remote builds in B7202: Diff 8877 for D1875: Remove status argument from methods!
Thu, Jan 23, 10:52 PM
themissinghlink updated the diff for D1874: Bay Bikes Weather Pipeline refactor.
  • tweaked test because c/i has no api key installed
  • renamed dagster_components to common
  • got rid of errant test file
Thu, Jan 23, 10:48 PM
themissinghlink added a comment to D1874: Bay Bikes Weather Pipeline refactor.

jesus christ good catch.....pytest refactor is not as smart as I thought.

Thu, Jan 23, 10:37 PM
prha accepted D1874: Bay Bikes Weather Pipeline refactor.

remove examples/dagster_examples_tests/dagster_components_tests/__init__.py?

Thu, Jan 23, 10:34 PM
sashank created D1875: Remove status argument from methods.
Thu, Jan 23, 10:20 PM
Harbormaster failed remote builds in B7197: Diff 8870 for D1863: [engine] shared step iteration!
Thu, Jan 23, 9:43 PM
themissinghlink added a comment to D1874: Bay Bikes Weather Pipeline refactor.

ready for a once over! Good call on the naming change and the container switch, this actually makes my general life a lot easier as well since I switched my dagster-home to point to the test-postgres-db.

Thu, Jan 23, 9:31 PM
schrockn added a comment to D1863: [engine] shared step iteration.

"risky" might be the wrong word but more like "too easy to break" meaning that anytime you change that config key it will certainly not behave the way you want to. Any change with the config would have to be coupled with a change to code. Given that it doesn't seem like something that should be in config

Thu, Jan 23, 9:21 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

Thu, Jan 23, 9:18 PM
themissinghlink updated the diff for D1874: Bay Bikes Weather Pipeline refactor.
  • renamed dagster_components to common and switched to using the test-postgres-db container
Thu, Jan 23, 9:17 PM
sashank committed R1:8b0966bce055: Fix example schedules (authored by sashank).
Fix example schedules
Thu, Jan 23, 9:17 PM
Sashank Thupukari <s@thupukari.com> closed D1865: Fix example schedules.
Thu, Jan 23, 9:17 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

So the crux of it is:

  • we do dagster/priority and it works for the core executors but (probably?) none of the others which leads to an unfortunate expectation mis-match in a not clearly communicated way (keys just ignored)
  • we do separate keys for each executor to align expectations ie dagster/in-process/priority, dagster-in-process/priority, whatever but thats aesthetically unfortunate and leads to having to set all keys to support different execution modes
Thu, Jan 23, 9:15 PM
schrockn added a comment to D1863: [engine] shared step iteration.

priority?

Thu, Jan 23, 9:09 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

what should the predefined keys be for the in process and multi process engines? I wasn't happy with what I came up with which motivated me to punt it out to config

Thu, Jan 23, 9:09 PM
max committed R1:4b2dbb31fe86: Working K8sRunLauncher (authored by max).
Working K8sRunLauncher
Thu, Jan 23, 8:46 PM
Max Gasner <max.gasner@gmail.com> closed D1873: Working K8sRunLauncher.
Thu, Jan 23, 8:46 PM
alangenfeld planned changes to D1853: handle differing arg and input def ordering.
Thu, Jan 23, 8:39 PM
Harbormaster failed remote builds in B7198: Diff 8871 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 8:37 PM
schrockn requested changes to D1863: [engine] shared step iteration.

So what I was expecting to see here is not a configurable priority key in terms of *configuring* executor on a per-run basis, but instead just having it be part of the executor definition, plus a comparator functions between priorities. Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

Thu, Jan 23, 8:35 PM
Harbormaster failed remote builds in B7199: Diff 8872 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 8:32 PM
max updated the diff for D1873: Working K8sRunLauncher.

Rebase

Thu, Jan 23, 8:26 PM
max updated the diff for D1873: Working K8sRunLauncher.

Nits

Thu, Jan 23, 8:23 PM
max updated the diff for D1873: Working K8sRunLauncher.

blk

Thu, Jan 23, 8:22 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

move sorting out to individual engines. use config to drive what key to use for prioirty on default engines

Thu, Jan 23, 8:18 PM
Harbormaster failed remote builds in B7196: Diff 8869 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 8:05 PM
max updated the diff for D1873: Working K8sRunLauncher.

py27

Thu, Jan 23, 7:51 PM
Harbormaster failed remote builds in B7195: Diff 8868 for D1867: [RFC] Remove schedule id!
Thu, Jan 23, 7:47 PM
Harbormaster failed remote builds in B7194: Diff 8867 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 7:40 PM
Harbormaster failed remote builds in B7193: Diff 8866 for D1873: Working K8sRunLauncher!
Thu, Jan 23, 7:37 PM
sashank updated the diff for D1867: [RFC] Remove schedule id.

rebase

Thu, Jan 23, 7:37 PM
prha added a comment to D1867: [RFC] Remove schedule id.

as in, not having the migration seems fine, when you can accomplish the same thing by running wipe and up

Thu, Jan 23, 7:28 PM
prha requested changes to D1867: [RFC] Remove schedule id.

I think it should be fine, but we should add something to the changelog, saying "Need to run dagster schedule ..."

Thu, Jan 23, 7:27 PM
max updated the diff for D1873: Working K8sRunLauncher.

Fixup py27

Thu, Jan 23, 7:26 PM
max updated the diff for D1873: Working K8sRunLauncher.

Up

Thu, Jan 23, 7:21 PM
max added inline comments to D1873: Working K8sRunLauncher.
Thu, Jan 23, 7:18 PM
sashank updated the diff for D1865: Fix example schedules.

Up

Thu, Jan 23, 7:13 PM
Harbormaster failed remote builds in B7191: Diff 8864 for D1865: Fix example schedules!
Thu, Jan 23, 7:02 PM
sashank updated the diff for D1865: Fix example schedules.

up

Thu, Jan 23, 6:53 PM
sashank committed R1:5b5055468c13: Move dagster_graphql_test context utils to dagster_graphql (authored by sashank).
Move dagster_graphql_test context utils to dagster_graphql
Thu, Jan 23, 6:51 PM