Page MenuHomePhabricator
Feed Advanced Search

Yesterday

alangenfeld commandeered D862: Add naive filesystem log watchers.
Wed, Aug 21, 10:43 PM
alangenfeld requested changes to D858: Overhauled run history view.
Wed, Aug 21, 10:27 PM
alangenfeld requested changes to D862: Add naive filesystem log watchers.

I will commandeer this (or the origin diff)

Wed, Aug 21, 9:50 PM
alangenfeld added inline comments to D862: Add naive filesystem log watchers.
Wed, Aug 21, 9:40 PM
alangenfeld accepted D859: Break log storage out from run storage.

move eventlog stuff to new file please

Wed, Aug 21, 9:35 PM
alangenfeld added a comment to D851: RFC: Add support for specifying Dask resources.

some other options to consider:

Wed, Aug 21, 9:10 PM
alangenfeld updated the summary of D861: [RFC] simpler user defined types section.
Wed, Aug 21, 8:59 PM
alangenfeld created D861: [RFC] simpler user defined types section.
Wed, Aug 21, 8:57 PM
alangenfeld accepted D853: Use __slots__ for PipelineRun.

no real cost to this though so whatevs

Wed, Aug 21, 3:20 PM
alangenfeld added a comment to D853: Use __slots__ for PipelineRun.

whats the point of slots? just speculative memory optimization?

Wed, Aug 21, 3:19 PM
alangenfeld accepted D852: Bring pyproject.toml black settings in line with CLI.
Wed, Aug 21, 3:10 PM
alangenfeld accepted D841: Consolidate double implementation of run storage.
Wed, Aug 21, 3:10 PM
alangenfeld accepted D850: Refactor PipelineRun.

the only thing to consider is moving some stuff out to separate files

Wed, Aug 21, 2:51 PM
alangenfeld committed R1:32f8b7f63ac5: [dagster] fix composition bug (authored by alangenfeld).
[dagster] fix composition bug
Wed, Aug 21, 2:47 PM
alangenfeld closed D847: [dagster] fix composition bug.
Wed, Aug 21, 2:47 PM

Tue, Aug 20

alangenfeld updated the diff for D847: [dagster] fix composition bug.

reuse solid handle to and use that to index for compute step

Tue, Aug 20, 9:00 PM
alangenfeld added a comment to D841: Consolidate double implementation of run storage.

just looking for answers to [1]

Tue, Aug 20, 8:14 PM
alangenfeld added a comment to D844: Initial pass at custom modification of the Dagre layout for wide pipelines.

do we have any tests for dagre?

Tue, Aug 20, 8:13 PM
alangenfeld accepted D845: Add dagster-aws up EC2 syncing.
Tue, Aug 20, 7:46 PM
alangenfeld requested changes to D842: Create ScheduleDefinition and dagster CLI scheduler commands.

dagster schedule list
dagster schedule running

Tue, Aug 20, 6:58 PM
alangenfeld updated the summary of D847: [dagster] fix composition bug.
Tue, Aug 20, 6:41 PM
alangenfeld updated the diff for D847: [dagster] fix composition bug.

cleamn

Tue, Aug 20, 6:41 PM
alangenfeld created D847: [dagster] fix composition bug.
Tue, Aug 20, 6:39 PM
alangenfeld created P17 (An Untitled Masterwork).
Tue, Aug 20, 3:14 PM
alangenfeld accepted D843: Break up AWS CLI and add tests.
Tue, Aug 20, 12:13 AM
alangenfeld added a comment to D843: Break up AWS CLI and add tests.

so this takes care of everything so its just pip install dagster-aws; dagster-aws; (include in test plan)?

Tue, Aug 20, 12:13 AM
alangenfeld accepted D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

thumbsup

Tue, Aug 20, 12:10 AM

Mon, Aug 19

alangenfeld added a comment to D841: Consolidate double implementation of run storage.

would love to see a stacked diff on top of this that takes a pass at cleaning up the weird class hierarchy / poor naming

Mon, Aug 19, 9:59 PM
alangenfeld added inline comments to D841: Consolidate double implementation of run storage.
Mon, Aug 19, 9:56 PM
alangenfeld changed the image for Image Macro "grumpycat".
Mon, Aug 19, 9:54 PM
alangenfeld created Image Macro "grumpycat".
Mon, Aug 19, 9:52 PM
alangenfeld committed R1:0de98e61b1d8: [dagster] whitelisted namedtuple serdes (authored by alangenfeld).
[dagster] whitelisted namedtuple serdes
Mon, Aug 19, 9:25 PM
alangenfeld closed D835: [dagster] whitelisted namedtuple serdes.
Mon, Aug 19, 9:25 PM
alangenfeld committed R1:c8b42fae28f9: [dagster] remove event_callback from RunConfig (authored by alangenfeld).
[dagster] remove event_callback from RunConfig
Mon, Aug 19, 9:24 PM
alangenfeld closed D829: [dagster] remove event_callback from RunConfig.
Mon, Aug 19, 9:24 PM
alangenfeld committed R1:8911ca4db53d: [dagster] EventSink (authored by alangenfeld).
[dagster] EventSink
Mon, Aug 19, 9:23 PM
alangenfeld closed D824: [dagster] EventSink.
Mon, Aug 19, 9:23 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

rebase

Mon, Aug 19, 9:15 PM
alangenfeld accepted D840: Temporarily pin ge to 0.7.6 or less.
Mon, Aug 19, 8:57 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

comment

Mon, Aug 19, 8:54 PM
alangenfeld accepted D832: Add EC2 setup script.

youve got enough code here to warrant breaking things out to a few files, now or in a subsequent diff

Mon, Aug 19, 7:48 PM
alangenfeld added a reviewer for D829: [dagster] remove event_callback from RunConfig: Restricted Project.

oops forgot to add reviewers

Mon, Aug 19, 6:44 PM
alangenfeld requested changes to D789: Naive watcher for filesystem log storage.

to your queue for restacking on some lower level clean ups

Mon, Aug 19, 6:40 PM
alangenfeld accepted D838: Re-enable coverage.

illallowit

Mon, Aug 19, 4:18 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

rebase

Mon, Aug 19, 3:47 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

up

Mon, Aug 19, 3:45 PM
alangenfeld accepted D830: Add initial sqllite run storage.
Mon, Aug 19, 3:30 PM
alangenfeld requested changes to D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.
Mon, Aug 19, 3:29 PM
alangenfeld added inline comments to D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.
Mon, Aug 19, 3:26 PM

Fri, Aug 16

alangenfeld added a comment to D789: Naive watcher for filesystem log storage.

ideally we also wouldn't even watch the log files for external runs unless we had an active subscription on them

Fri, Aug 16, 11:35 PM
alangenfeld added a comment to D789: Naive watcher for filesystem log storage.

we could land this after a bit of cleanup - but i think im tempted to just have this feed in to a more systemic refactor of all this PipelineRun stuff.

Fri, Aug 16, 11:13 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

open and close connections every time

Fri, Aug 16, 9:52 PM
alangenfeld added inline comments to D830: Add initial sqllite run storage.
Fri, Aug 16, 9:39 PM
alangenfeld resigned from D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

defering to max
cleanup

Fri, Aug 16, 9:36 PM
alangenfeld added a comment to D832: Add EC2 setup script.

should put this somewhere other than bin?

Fri, Aug 16, 9:35 PM
alangenfeld updated subscribers of D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

looks pretty solid to me

Fri, Aug 16, 9:30 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

hold connection

Fri, Aug 16, 9:22 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

rebase

Fri, Aug 16, 8:54 PM
alangenfeld updated the diff for D829: [dagster] remove event_callback from RunConfig.

rebase

Fri, Aug 16, 8:52 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

rebase

Fri, Aug 16, 8:52 PM
alangenfeld committed R1:981cb1cb36ec: [dagster-graphql] use obj for required resource (authored by alangenfeld).
[dagster-graphql] use obj for required resource
Fri, Aug 16, 8:44 PM
alangenfeld closed D836: [dagster-graphql] use obj for required resource.
Fri, Aug 16, 8:44 PM
alangenfeld accepted D821: Add dbt test solid.

might be worth moving from init to a couple smaller files

Fri, Aug 16, 8:38 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

rebase w/ fixes

Fri, Aug 16, 8:31 PM
alangenfeld created D836: [dagster-graphql] use obj for required resource.
Fri, Aug 16, 8:30 PM
alangenfeld updated the diff for D835: [dagster] whitelisted namedtuple serdes.

use decorator

Fri, Aug 16, 6:50 PM
alangenfeld updated subscribers of D835: [dagster] whitelisted namedtuple serdes.
Fri, Aug 16, 6:06 PM
alangenfeld planned changes to D835: [dagster] whitelisted namedtuple serdes.
Fri, Aug 16, 5:46 PM
alangenfeld created D835: [dagster] whitelisted namedtuple serdes.
Fri, Aug 16, 5:30 PM

Thu, Aug 15

alangenfeld updated the diff for D829: [dagster] remove event_callback from RunConfig.

fixes

Thu, Aug 15, 11:26 PM
alangenfeld created D829: [dagster] remove event_callback from RunConfig.
Thu, Aug 15, 10:17 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

comments

Thu, Aug 15, 9:24 PM
alangenfeld added inline comments to D824: [dagster] EventSink.
Thu, Aug 15, 9:06 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

rebase

Thu, Aug 15, 9:00 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

fixe

Thu, Aug 15, 8:26 PM
alangenfeld planned changes to D824: [dagster] EventSink.
Thu, Aug 15, 7:05 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

add on_pipeline_init and on_pipeline_teardown methods

Thu, Aug 15, 6:56 PM
alangenfeld updated the summary of D824: [dagster] EventSink.
Thu, Aug 15, 6:40 PM
alangenfeld updated the diff for D824: [dagster] EventSink.

clean up

Thu, Aug 15, 6:36 PM
alangenfeld created D824: [dagster] EventSink.
Thu, Aug 15, 6:25 PM
alangenfeld added a comment to D819: Add custom behavior to the CellTruncationProvider.

this looks good to me but ill give @bengotow time to look

Thu, Aug 15, 3:55 PM

Wed, Aug 14

alangenfeld committed R1:fb384ad3b262: [log sink] multiprocess executor (authored by alangenfeld).
[log sink] multiprocess executor
Wed, Aug 14, 8:41 PM
alangenfeld closed D812: [log sink] multiprocess executor.
Wed, Aug 14, 8:41 PM
alangenfeld added inline comments to D815: Promote some step events to INFO.
Wed, Aug 14, 8:28 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

cant remove the log_step_event call

Wed, Aug 14, 8:12 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

fixes

Wed, Aug 14, 7:22 PM
alangenfeld added a comment to D815: Promote some step events to INFO.

what does dagster cli output look like on hello world?

Wed, Aug 14, 6:49 PM
alangenfeld accepted D813: Throwing up example yaml-driven deps.

can you throw a big comment block somewhere in here indicating exactly what this is? I could see some one hitting this too early and thinking its "the way" to do things

Wed, Aug 14, 6:48 PM
alangenfeld accepted D804: Create Scheduler GraphQL interface.
Wed, Aug 14, 6:46 PM
alangenfeld closed D811: [log sink] RunConfig log_sink.
Wed, Aug 14, 6:43 PM
alangenfeld committed R1:b4341bc52150: [log sink] RunConfig log_sink (authored by alangenfeld).
[log sink] RunConfig log_sink
Wed, Aug 14, 6:43 PM
alangenfeld committed R1:6bd22b24048f: [log sink] move dagstermill xproc log to dagster (authored by alangenfeld).
[log sink] move dagstermill xproc log to dagster
Wed, Aug 14, 6:43 PM
alangenfeld closed D810: [log sink] move dagstermill xproc log to dagster.
Wed, Aug 14, 6:43 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

rm spurious changes

Wed, Aug 14, 6:42 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

open connection every write

Wed, Aug 14, 6:41 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

try/finally & faster poll rate

Wed, Aug 14, 4:12 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

lint

Wed, Aug 14, 3:06 PM
alangenfeld updated the diff for D811: [log sink] RunConfig log_sink.

fix dask

Wed, Aug 14, 3:05 PM
alangenfeld updated the diff for D812: [log sink] multiprocess executor.

rebase

Wed, Aug 14, 2:54 PM
alangenfeld updated the diff for D811: [log sink] RunConfig log_sink.

fix bad callsite

Wed, Aug 14, 2:50 PM