Page MenuHomePhabricator
Feed All Stories

Today

max updated the summary of D2076: Add exponential backoff utility.
Thu, Feb 20, 1:07 AM
max updated the diff for D2076: Add exponential backoff utility.

Disable dask tests again

Thu, Feb 20, 1:07 AM
Harbormaster failed remote builds in B7933: Diff 9792 for D2061: [dagster-k8s] Remove hardcoded dagster home!
Thu, Feb 20, 1:07 AM
max created D2080: Don't use the Alembic EnvironmentContext.
Thu, Feb 20, 1:04 AM
themissinghlink updated the diff for D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.
  • got rid of silly name keyword param and made it required
Thu, Feb 20, 12:29 AM
schrockn added a comment to D2073: change scoped pipeline context to event generator.

I'm quite happy with this direction. curious what you think @alangenfeld

Thu, Feb 20, 12:27 AM
schrockn added a comment to D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.
>>> import dagster_pandas
from >>> from dagster_pandas import create_dagster_pandas_dataframe_type
>>> create_dagster_pandas_dataframe_type(name=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/schrockn/code/dagster/python_modules/libraries/dagster-pandas/dagster_pandas/data_frame.py", line 186, in create_dagster_pandas_dataframe_type
    description=description,
  File "/Users/schrockn/code/dagster/python_modules/dagster/dagster/core/types/dagster_type.py", line 86, in __init__
    check.invariant(not (name is None and key is None), 'Must set key or name')
  File "/Users/schrockn/code/dagster/python_modules/dagster/dagster/check/__init__.py", line 166, in invariant
    CheckError('Invariant failed. Description: {desc}'.format(desc=desc))
  File "/Users/schrockn/venvs/dagster-dev-3.7.4/lib/python3.7/site-packages/future/utils/__init__.py", line 419, in raise_with_traceback
    raise exc.with_traceback(traceback)
dagster.check.CheckError: Invariant failed. Description: Must set key or name
>>>
Thu, Feb 20, 12:24 AM
schrockn requested changes to D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.

still need to make name required. name=None will throw. a test should confirm this

Thu, Feb 20, 12:23 AM
schrockn added inline comments to D2073: change scoped pipeline context to event generator.
Thu, Feb 20, 12:17 AM
themissinghlink updated the diff for D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.
  • made docs fixes and added comment
Thu, Feb 20, 12:14 AM
prha added a comment to D2073: change scoped pipeline context to event generator.

couple things.... still trying to figure out the exception handling. I think we weren't tearing down resources properly before, so figuring out what tests we need to write, change, etc.

Thu, Feb 20, 12:12 AM
prha updated the diff for D2073: change scoped pipeline context to event generator.
  • pipeline initialization events, but with generators INSTEAD of context managers
Thu, Feb 20, 12:06 AM
themissinghlink added inline comments to D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.
Thu, Feb 20, 12:04 AM
nate planned changes to D2061: [dagster-k8s] Remove hardcoded dagster home.

just getting this off my laptop before commute, will update further tomorrow

Thu, Feb 20, 12:04 AM
nate updated the diff for D2061: [dagster-k8s] Remove hardcoded dagster home.

WIP on discussed changes

Thu, Feb 20, 12:03 AM
schrockn requested changes to D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.

req'ing changes based on name being optional

Thu, Feb 20, 12:00 AM

Yesterday

schrockn added a comment to D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types..

would recommend using arc patch in that situation

Wed, Feb 19, 11:58 PM
themissinghlink added a reviewer for D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX: schrockn.
Wed, Feb 19, 11:46 PM
themissinghlink abandoned D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types..

Holy shit. In all the excitement, I realized I never checked out a branch....uhhhh gonna abandon this and get a new revision out with the fixes. I'll link this revision to that one but will need another LGTM

Wed, Feb 19, 11:42 PM
themissinghlink created D2079: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types REDUX.
Wed, Feb 19, 11:42 PM
max added a comment to D2076: Add exponential backoff utility.

I'd prefer not to take the dependency but will defer to others

Wed, Feb 19, 11:30 PM
themissinghlink updated the diff for D2060: make dagster pandas input/output schema more flexible.
  • rebasing to pick up buildkite changes.
Wed, Feb 19, 11:09 PM
themissinghlink committed R1:bb22e5245081: Refactor generate_training_set pipeline. (authored by themissinghlink).
Refactor generate_training_set pipeline.
Wed, Feb 19, 11:05 PM
themissinghlink closed D2071: Refactor generate_training_set pipeline..
Wed, Feb 19, 11:05 PM
themissinghlink added inline comments to D2071: Refactor generate_training_set pipeline..
Wed, Feb 19, 11:02 PM
Harbormaster failed remote builds in B7929: Diff 9787 for D2060: make dagster pandas input/output schema more flexible!
Wed, Feb 19, 10:59 PM
schrockn added a comment to D2071: Refactor generate_training_set pipeline..

cool

Wed, Feb 19, 10:54 PM
themissinghlink added a comment to D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types..

On it.

Wed, Feb 19, 10:54 PM
schrockn accepted D2071: Refactor generate_training_set pipeline..
Wed, Feb 19, 10:52 PM
schrockn accepted D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types..

looks good. plz heed final comments!

Wed, Feb 19, 10:48 PM
schrockn added inline comments to D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types..
Wed, Feb 19, 10:47 PM
Harbormaster failed remote builds in B7928: Diff 9786 for D2061: [dagster-k8s] Remove hardcoded dagster home!
Wed, Feb 19, 10:35 PM
nate added inline comments to D2061: [dagster-k8s] Remove hardcoded dagster home.
Wed, Feb 19, 10:31 PM
nate added a comment to D2061: [dagster-k8s] Remove hardcoded dagster home.

yeah we could do something like that, although for secrets we'll use an entirely different k8s object, Secret https://kubernetes.io/docs/concepts/configuration/secret/ instead of a ConfigMap—so instead of the previous approach, we'd do like DAGSTER_PG_PASSWORD above—see [1]

Wed, Feb 19, 10:30 PM
alangenfeld added a comment to D2061: [dagster-k8s] Remove hardcoded dagster home.

is this env: approach going to support loading secrets in as environment variables? The the env_from approach supports that right? Should we have a base then dagit and job_runner specific env_from that defaults to something populated by the env: approach? That way the overriding behavior is handled by env_from load order on the pod?

Wed, Feb 19, 10:26 PM
themissinghlink added a reviewer for D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types.: nate.
Wed, Feb 19, 10:17 PM
nate added a comment to D2061: [dagster-k8s] Remove hardcoded dagster home.

have an explicit base and then the two override sections

yeah what I'm suggesting is just to have the base map of env vars and drop support for overrides; the dagit/job-specific overrides _can_ work, but e.g. if I have:

Wed, Feb 19, 10:16 PM
alangenfeld added a comment to D2061: [dagster-k8s] Remove hardcoded dagster home.

We could have one global Dagster ConfigMap to spec extra env vars for everywhere (dagit, celery workers, job containers)—or, as in this diff, separate env ConfigMaps for dagit vs. celery/job nodes. I lean towards the former to keep things simple, but lmk

Wed, Feb 19, 10:05 PM
alangenfeld created D2078: fix event log pagination.
Wed, Feb 19, 9:58 PM
themissinghlink updated the diff for D2060: make dagster pandas input/output schema more flexible.
  • renamed to more explicit val
Wed, Feb 19, 9:55 PM
nate added a comment to D2061: [dagster-k8s] Remove hardcoded dagster home.

@alangenfeld RFC here

Wed, Feb 19, 9:40 PM
alangenfeld added inline comments to D1471: Refactor SQL-based event log storages to use SQLAlchemy.
Wed, Feb 19, 9:32 PM
nate updated the diff for D2061: [dagster-k8s] Remove hardcoded dagster home.

up

Wed, Feb 19, 9:31 PM
Harbormaster failed remote builds in B7926: Diff 9784 for D2076: Add exponential backoff utility!
Wed, Feb 19, 9:20 PM
nate accepted D2076: Add exponential backoff utility.

Looks reasonable to me; only question I have here is whether we should just use tenacity (https://tenacity.readthedocs.io/en/latest/) vs. rolling our own—thoughts?

Wed, Feb 19, 9:10 PM
Harbormaster failed remote builds in B7925: Diff 9783 for D2076: Add exponential backoff utility!
Wed, Feb 19, 9:10 PM
themissinghlink added a comment to D2060: make dagster pandas input/output schema more flexible.

Regardless of whether pandas' API changes in the future, we need to be compatible with all versions of pandas which makes it difficult to have a strictly typed schema right? I mean it's a tradeoff, but I don't know if we want to tie ourselves to maintaining it.

Wed, Feb 19, 9:04 PM
themissinghlink updated the diff for D2071: Refactor generate_training_set pipeline..
  • addressed feedback
Wed, Feb 19, 8:46 PM
themissinghlink added inline comments to D2071: Refactor generate_training_set pipeline..
Wed, Feb 19, 8:46 PM
Harbormaster failed remote builds in B7921: Diff 9778 for D2076: Add exponential backoff utility!
Wed, Feb 19, 8:44 PM
themissinghlink added reviewers for D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types.: schrockn, max.
Wed, Feb 19, 8:36 PM
GitHub <noreply@github.com> committed R1:280cf434f93e: feat: add new docs header and some fixes (#2171) (authored by Pedro Nauck <pedronauck@gmail.com>).
feat: add new docs header and some fixes (#2171)
Wed, Feb 19, 8:28 PM
max added a comment to D2060: make dagster pandas input/output schema more flexible.

Pandas is at 1.0 now -- https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.0.0.html

Wed, Feb 19, 8:26 PM
max updated the summary of D2076: Add exponential backoff utility.
Wed, Feb 19, 8:17 PM
max updated the diff for D2076: Add exponential backoff utility.

Up

Wed, Feb 19, 8:16 PM
max updated the diff for D2076: Add exponential backoff utility.

Six

Wed, Feb 19, 8:06 PM
max committed R1:08b7918f8629: Rework sidebar styles (authored by max).
Rework sidebar styles
Wed, Feb 19, 7:59 PM
max closed D2064: Rework sidebar styles.
Wed, Feb 19, 7:59 PM
Harbormaster failed remote builds in B7923: Diff 9780 for D2076: Add exponential backoff utility!
Wed, Feb 19, 7:54 PM
max updated the diff for D2064: Rework sidebar styles.

Rebase

Wed, Feb 19, 7:42 PM
max updated the diff for D2076: Add exponential backoff utility.

Resolves https://github.com/dagster-io/dagster/issues/2174

Wed, Feb 19, 7:42 PM
themissinghlink created D2077: Make input_hydration_configs and output_materialization_configs parametrizable for custom dagster dataframe types..
Wed, Feb 19, 7:41 PM
max created D2076: Add exponential backoff utility.
Wed, Feb 19, 7:40 PM
alangenfeld committed R1:054cdc244c96: ActiveExecution cleanup (authored by alangenfeld).
ActiveExecution cleanup
Wed, Feb 19, 7:11 PM
alangenfeld closed D2067: ActiveExecution cleanup.
Wed, Feb 19, 7:11 PM
prha updated the diff for D2073: change scoped pipeline context to event generator.

fix scoped_pipeline_context reference

Wed, Feb 19, 6:49 PM
schrockn added inline comments to D2073: change scoped pipeline context to event generator.
Wed, Feb 19, 6:48 PM
Harbormaster failed remote builds in B7916: Diff 9770 for D2069: Check for run when resolving schedule attempt!
Wed, Feb 19, 6:45 PM
sashank committed R1:d6c762e7c00b: Check for run when resolving schedule attempt (authored by sashank).
Check for run when resolving schedule attempt
Wed, Feb 19, 6:28 PM
sashank closed D2069: Check for run when resolving schedule attempt.
Wed, Feb 19, 6:28 PM
Harbormaster failed remote builds in B7919: Diff 9774 for D2073: change scoped pipeline context to event generator!
Wed, Feb 19, 6:27 PM
prha updated the diff for D2073: change scoped pipeline context to event generator.

add try/finally, remove double defaults

Wed, Feb 19, 6:18 PM
prha added a comment to D2073: change scoped pipeline context to event generator.

I initially had it so that the resource/builder/context were yielded along with the event stream, but switched it (per @alangenfeld's suggestion) because it makes it way more explicit in the caller what is going on. I do like this better and we're not mixing types so much. It was getting confusing where we're stacking things at different levels and the type I'm trying to extract is different at each level.

Wed, Feb 19, 6:10 PM
schrockn requested changes to D2071: Refactor generate_training_set pipeline..

cool stuff. like how hard you are pushing on modes/testing

Wed, Feb 19, 6:10 PM
prha committed R1:092691b676ec: fix reload supported flag on dagit reloader (authored by prha).
fix reload supported flag on dagit reloader
Wed, Feb 19, 5:58 PM
prha closed D2075: fix reload supported flag on dagit reloader.
Wed, Feb 19, 5:58 PM
schrockn requested changes to D2073: change scoped pipeline context to event generator.

This seems like a good direction. Great job wading through this. Thorny part of the codebase for sure.

Wed, Feb 19, 5:57 PM
alangenfeld updated the diff for D2067: ActiveExecution cleanup.

rebase

Wed, Feb 19, 5:53 PM
sashank updated the diff for D2069: Check for run when resolving schedule attempt.

Rebase

Wed, Feb 19, 5:45 PM
sashank updated the diff for D2069: Check for run when resolving schedule attempt.

Move to branch

Wed, Feb 19, 5:41 PM
alangenfeld added a reviewer for D2070: [dagit] surface Failure metadata: sashank.
Wed, Feb 19, 5:24 PM
alangenfeld updated the diff for D2070: [dagit] surface Failure metadata.

rebase

Wed, Feb 19, 5:24 PM
alangenfeld accepted D2069: Check for run when resolving schedule attempt.

illallowit

Wed, Feb 19, 5:17 PM
sashank accepted D2072: [dagit] improve event log tags.

Nice improvement 👍

Wed, Feb 19, 4:57 PM
alangenfeld accepted D2075: fix reload supported flag on dagit reloader.
Wed, Feb 19, 3:59 PM
alangenfeld created Image Macro "ooof".
Wed, Feb 19, 3:59 PM
alangenfeld updated the diff for D2072: [dagit] improve event log tags.

rebase

Wed, Feb 19, 3:54 PM
themissinghlink added a reviewer for D2071: Refactor generate_training_set pipeline.: nate.
Wed, Feb 19, 4:02 AM
themissinghlink added reviewers for D2071: Refactor generate_training_set pipeline.: schrockn, alangenfeld.
Wed, Feb 19, 4:02 AM
themissinghlink updated the summary of D2071: Refactor generate_training_set pipeline..
Wed, Feb 19, 3:52 AM
themissinghlink updated the diff for D2071: Refactor generate_training_set pipeline..
  • brought in LocalClient resource
Wed, Feb 19, 3:48 AM
prha accepted D2067: ActiveExecution cleanup.
Wed, Feb 19, 12:45 AM
Harbormaster failed remote builds in B7909: Diff 9763 for D2072: [dagit] improve event log tags!
Wed, Feb 19, 12:32 AM
prha updated the diff for D2075: fix reload supported flag on dagit reloader.

rebase

Wed, Feb 19, 12:32 AM
Harbormaster failed remote builds in B7905: Diff 9758 for D1883: warmup!
Wed, Feb 19, 12:32 AM
Harbormaster failed remote builds in B7911: Diff 9765 for D2075: fix reload supported flag on dagit reloader!
Wed, Feb 19, 12:31 AM
themissinghlink added a comment to D2071: Refactor generate_training_set pipeline..

Done because I realized I need to make a core change.

Wed, Feb 19, 12:24 AM
themissinghlink removed reviewers for D2071: Refactor generate_training_set pipeline.: schrockn, nate, alangenfeld.
Wed, Feb 19, 12:24 AM
prha created D2075: fix reload supported flag on dagit reloader.
Wed, Feb 19, 12:11 AM
themissinghlink updated the diff for D2071: Refactor generate_training_set pipeline..
  • rebased with master to pick up buildkite changes
Wed, Feb 19, 12:05 AM