Page MenuHomePhabricator
Feed All Stories

Today

natekupp updated the diff for D832: Add EC2 setup script.

add shell command

Mon, Aug 19, 2:10 AM

Yesterday

natekupp updated the diff for D832: Add EC2 setup script.

up

Sun, Aug 18, 10:29 PM
natekupp updated the diff for D832: Add EC2 setup script.

up

Sun, Aug 18, 10:25 PM
natekupp updated the test plan for D832: Add EC2 setup script.
Sun, Aug 18, 9:23 PM
schrockn added inline comments to D830: Add initial sqllite run storage.
Sun, Aug 18, 7:24 PM
schrockn updated the diff for D830: Add initial sqllite run storage.

conn instead of provider

Sun, Aug 18, 6:46 PM

Sat, Aug 17

schrockn added inline comments to D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.
Sat, Aug 17, 2:41 PM
natekupp created D837: Re-enable coveralls.
Sat, Aug 17, 12:25 AM
natekupp updated the diff for D833: Update Flask dependency.

up

Sat, Aug 17, 12:09 AM

Fri, Aug 16

alangenfeld added a comment to D789: Naive watcher for filesystem log storage.

ideally we also wouldn't even watch the log files for external runs unless we had an active subscription on them

Fri, Aug 16, 11:35 PM
Harbormaster failed remote builds in B3038: Diff 3795 for D833: Update Flask dependency!
Fri, Aug 16, 11:22 PM
natekupp updated the diff for D833: Update Flask dependency.

up

Fri, Aug 16, 11:16 PM
max updated the diff for D789: Naive watcher for filesystem log storage.

Fixup

Fri, Aug 16, 11:14 PM
alangenfeld added a comment to D789: Naive watcher for filesystem log storage.

we could land this after a bit of cleanup - but i think im tempted to just have this feed in to a more systemic refactor of all this PipelineRun stuff.

Fri, Aug 16, 11:13 PM
Harbormaster failed remote builds in B3036: Diff 3793 for D789: Naive watcher for filesystem log storage!
Fri, Aug 16, 11:12 PM
max updated the diff for D789: Naive watcher for filesystem log storage.

Nits

Fri, Aug 16, 11:05 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Add mock to dagster-graphql dev requirements

Fri, Aug 16, 10:54 PM
Harbormaster failed remote builds in B3034: Diff 3791 for D789: Naive watcher for filesystem log storage!
Fri, Aug 16, 10:51 PM
max updated the diff for D789: Naive watcher for filesystem log storage.

Nits

Fri, Aug 16, 10:45 PM
Harbormaster failed remote builds in B3033: Diff 3790 for D789: Naive watcher for filesystem log storage!
Fri, Aug 16, 10:42 PM
max updated the diff for D789: Naive watcher for filesystem log storage.

nk

Fri, Aug 16, 10:36 PM
natekupp added a comment to D833: Update Flask dependency.
(venv) root@ip-172-31-1-11:/# dagit
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (Flask 1.1.1 (/venv/lib/python3.5/site-packages), Requirement.parse('Flask<1.1.0,>=0.12.4'), {'dagit'})
Fri, Aug 16, 10:20 PM
max accepted D824: [dagster] EventSink.
Fri, Aug 16, 10:17 PM
Harbormaster failed remote builds in B3032: Diff 3789 for D789: Naive watcher for filesystem log storage!
Fri, Aug 16, 10:13 PM
max accepted D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.
Fri, Aug 16, 10:13 PM
max added reviewers for D789: Naive watcher for filesystem log storage: alangenfeld, Restricted Project.
Fri, Aug 16, 10:08 PM
max retitled D789: Naive watcher for filesystem log storage from Wip to Naive watcher for filesystem log storage.
Fri, Aug 16, 10:08 PM
Harbormaster failed remote builds in B3031: Diff 3788 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 10:07 PM
max updated the diff for D789: Naive watcher for filesystem log storage.

Nits

Fri, Aug 16, 10:07 PM
Harbormaster failed remote builds in B3030: Diff 3785 for D789: Naive watcher for filesystem log storage!
Fri, Aug 16, 10:02 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

rebase

Fri, Aug 16, 10:01 PM
schrockn committed R1:69f8dcb78787: Add dbt test solid (authored by schrockn).
Add dbt test solid
Fri, Aug 16, 9:59 PM
schrockn closed D821: Add dbt test solid.
Fri, Aug 16, 9:59 PM
Harbormaster failed remote builds in B3029: Diff 3784 for D835: [dagster] whitelisted namedtuple serdes!
Fri, Aug 16, 9:58 PM
max committed R1:ed69837efeba: Update releasing docs (authored by max).
Update releasing docs
Fri, Aug 16, 9:57 PM
Max Gasner <max.gasner@gmail.com> closed D828: Update releasing docs.
Fri, Aug 16, 9:57 PM
max added a comment to D833: Update Flask dependency.

Do we know what is actually breaking?

Fri, Aug 16, 9:57 PM
max updated the diff for D789: Naive watcher for filesystem log storage.

POC

Fri, Aug 16, 9:56 PM
sashank added inline comments to D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.
Fri, Aug 16, 9:55 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

open and close connections every time

Fri, Aug 16, 9:52 PM
Harbormaster failed remote builds in B3027: Diff 3782 for D835: [dagster] whitelisted namedtuple serdes!
Fri, Aug 16, 9:48 PM
Harbormaster failed remote builds in B3028: Diff 3783 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 9:46 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Remove use of kwargs in start and end schedule mutations

Fri, Aug 16, 9:40 PM
alangenfeld added inline comments to D830: Add initial sqllite run storage.
Fri, Aug 16, 9:39 PM
alangenfeld resigned from D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

defering to max
cleanup

Fri, Aug 16, 9:36 PM
alangenfeld added a comment to D832: Add EC2 setup script.

should put this somewhere other than bin?

Fri, Aug 16, 9:35 PM
alangenfeld updated subscribers of D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

looks pretty solid to me

Fri, Aug 16, 9:30 PM
Harbormaster failed remote builds in B3026: Diff 3781 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 9:25 PM
Harbormaster failed remote builds in B3025: Diff 3780 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 9:22 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

hold connection

Fri, Aug 16, 9:22 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Add bash script deletion check in test_start_and_end_schedule()

Fri, Aug 16, 9:19 PM
sashank retitled D826: Implement start_schedule and end_schedule for SystemCronTabScheduler from Implement crontab scheduler start and end to Implement start_schedule and end_schedule for SystemCronTabScheduler.
Fri, Aug 16, 9:16 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Mock $DAGSTER_HOME for test

Fri, Aug 16, 9:15 PM
Harbormaster failed remote builds in B3022: Diff 3777 for D835: [dagster] whitelisted namedtuple serdes!
Fri, Aug 16, 9:13 PM
sashank retitled D826: Implement start_schedule and end_schedule for SystemCronTabScheduler from [WIP] Implement crontab scheduler start and end to Implement crontab scheduler start and end.
Fri, Aug 16, 9:08 PM
Harbormaster failed remote builds in B3024: Diff 3779 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 9:02 PM
Harbormaster failed remote builds in B3023: Diff 3778 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 9:01 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

make graphql

Fri, Aug 16, 8:55 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Fix end_schedule, and remove hardcoded DAGSTER_HOME

Fri, Aug 16, 8:55 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

rebase

Fri, Aug 16, 8:54 PM
alangenfeld updated the diff for D829: [dagster] remove event_callback from RunConfig.

rebase

Fri, Aug 16, 8:52 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

rebase

Fri, Aug 16, 8:52 PM
Harbormaster failed remote builds in B3019: Diff 3773 for D835: [dagster] whitelisted namedtuple serdes!
Fri, Aug 16, 8:45 PM
alangenfeld committed R1:981cb1cb36ec: [dagster-graphql] use obj for required resource (authored by alangenfeld).
[dagster-graphql] use obj for required resource
Fri, Aug 16, 8:44 PM
alangenfeld closed D836: [dagster-graphql] use obj for required resource.
Fri, Aug 16, 8:44 PM
Harbormaster failed remote builds in B3018: Diff 3772 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 8:39 PM
alangenfeld accepted D821: Add dbt test solid.

might be worth moving from init to a couple smaller files

Fri, Aug 16, 8:38 PM
prha accepted D836: [dagster-graphql] use obj for required resource.
Fri, Aug 16, 8:38 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

rebase w/ fixes

Fri, Aug 16, 8:31 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Write to log file with filename {schedule name}_{schedule id}.log in the same directory as the bash script and metadata file

Fri, Aug 16, 8:31 PM
Harbormaster failed remote builds in B3016: Diff 3770 for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler!
Fri, Aug 16, 8:31 PM
alangenfeld created D836: [dagster-graphql] use obj for required resource.
Fri, Aug 16, 8:30 PM
sashank updated the diff for D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Mutations for start and end schedule, create TestSystemCronScheduler, and write tests for all new queries and mutations

Fri, Aug 16, 8:18 PM
Harbormaster failed remote builds in B3015: Diff 3769 for D835: [dagster] whitelisted namedtuple serdes!
Fri, Aug 16, 7:36 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

use decorator

Fri, Aug 16, 6:50 PM
Harbormaster failed remote builds in B3014: Diff 3768 for D835: [dagster] whitelisted namedtuple serdes!
Fri, Aug 16, 6:36 PM
alangenfeld updated subscribers of D835: [dagster] whitelisted namedtuple serdes.
Fri, Aug 16, 6:06 PM
alangenfeld planned changes to D835: [dagster] whitelisted namedtuple serdes.
Fri, Aug 16, 5:46 PM
alangenfeld created D835: [dagster] whitelisted namedtuple serdes.
Fri, Aug 16, 5:30 PM
schrockn added a comment to D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

lovely

Fri, Aug 16, 4:26 PM
prha added a comment to D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

Fri, Aug 16, 3:49 PM
natekupp updated subscribers of D833: Update Flask dependency.

It's a good question - I'm not sure if there's any good way to solve this, maybe @max has some thoughts since he was looking at the airflow <> flask stuff originally?

Fri, Aug 16, 3:06 PM
schrockn added a comment to D833: Update Flask dependency.

So will limit what versions of airflow we can integrate with? I’m a little hazy on how this all coheres to a flexible sane strategy around deps

Fri, Aug 16, 2:57 PM
natekupp updated the diff for D833: Update Flask dependency.

up

Fri, Aug 16, 2:48 PM
schrockn updated subscribers of D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

This looks great to me. I'll let @max approve to resolve your question re: the use/non-use of raise_from.

Fri, Aug 16, 1:41 PM
schrockn accepted D833: Update Flask dependency.
Fri, Aug 16, 1:38 PM
schrockn updated the diff for D821: Add dbt test solid.

up

Fri, Aug 16, 1:36 PM
prha committed R1:466c4aaab81e: fix airline demo snippet (authored by prha).
fix airline demo snippet
Fri, Aug 16, 5:10 AM
prha closed D834: fix airline demo snippet.
Fri, Aug 16, 5:10 AM
natekupp accepted D834: fix airline demo snippet.
Fri, Aug 16, 1:10 AM
prha created D834: fix airline demo snippet.
Fri, Aug 16, 12:33 AM
prha committed R1:b60d6f85d125: adjust the dagstermill code snippets for the airline demo docs (authored by prha).
adjust the dagstermill code snippets for the airline demo docs
Fri, Aug 16, 12:05 AM
prha closed D823: adjust the dagstermill code snippets for the airline demo docs.
Fri, Aug 16, 12:05 AM
natekupp accepted D823: adjust the dagstermill code snippets for the airline demo docs.

yeah its definitely brittle. An alternative we've considered in the past is to just inline the code, but then the code samples are not tested. Definitely open to ideas if you've seen this done any other/better way

Fri, Aug 16, 12:02 AM

Thu, Aug 15

natekupp added a reviewer for D833: Update Flask dependency: Restricted Project.
Thu, Aug 15, 11:54 PM
prha added a reviewer for D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`: Restricted Project.
Thu, Aug 15, 11:52 PM
natekupp updated the summary of D832: Add EC2 setup script.
Thu, Aug 15, 11:50 PM
natekupp added a comment to D832: Add EC2 setup script.

Looks like this:

Thu, Aug 15, 11:48 PM
natekupp updated the diff for D832: Add EC2 setup script.

up

Thu, Aug 15, 11:46 PM
prha removed a reviewer for D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`: Restricted Project.
Thu, Aug 15, 11:44 PM