Page MenuHomePhabricator

nate (Nate Kupp)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Apr 23 2019, 10:22 PM (83 w, 4 d)
Roles
Administrator

Recent Activity

Wed, Nov 25

nate abandoned D4565: Add more info to pipeline run status checks.

gonna abandon this for now, can come back to it if needed

Wed, Nov 25, 12:54 AM

Tue, Nov 24

nate added a comment to D5264: Remove py2 from docker images.

yesss

Tue, Nov 24, 5:25 PM
nate updated subscribers of D5259: Unpin kubernetes in dagster-k8s.

hmmm this doesn't seem to have triggered the integration test suite, if I'm not mistaken - @max ?

Tue, Nov 24, 4:35 AM
nate accepted D5040: use AssetStore in basic pyspark example.

make_people() has gotten so much mileage ;)

Tue, Nov 24, 3:13 AM

Sat, Nov 21

nate abandoned D2259: RFC: Add HDFS system storage.

Abandoning given incoming changes to this part of the system. We can revisit in the future if needed cc @sandyryza @yuhan

Sat, Nov 21, 12:25 AM

Fri, Nov 20

nate resigned from D3400: (RFC) scaffolding for validation/quality classes.
Fri, Nov 20, 6:24 PM

Wed, Nov 18

nate resigned from D5177: [dagit] Fix schedule name truncation in left nav.

looks reasonable enough to me, but will let ben/prha do the signoff :)

Wed, Nov 18, 6:19 PM

Tue, Nov 17

nate accepted D5131: Prevent k8s images from sending telemetry.

cool sgtm!

Tue, Nov 17, 7:41 PM

Mon, Nov 16

nate abandoned D2796: WIP on case class schema inference.
Mon, Nov 16, 4:07 PM

Fri, Nov 13

nate added a comment to D5131: Prevent k8s images from sending telemetry.

rather than adding the buildkite env var here, should we also add support for blocking telemetry based on a DAGSTER_TELEMETRY_ENABLED=false env var? seems like something our users might want also

Fri, Nov 13, 2:40 PM

Wed, Nov 11

nate accepted D5099: change postgres default retry attempts.
Wed, Nov 11, 9:02 PM

Tue, Nov 10

nate accepted D5087: GCSComputeLogManager.

see one comment, otherwise lgtm

Tue, Nov 10, 11:08 PM

Mon, Nov 9

nate accepted D5068: Test integration trigger.
Mon, Nov 9, 8:13 PM

Mon, Nov 2

nate accepted D4883: Break integration tests out into separate Python file.

See one nit above, otherwise LGTM.

Mon, Nov 2, 8:10 PM

Sun, Nov 1

nate added inline comments to D4984: Add retries to sqlite event log connection, ensure event log always tries to connect at least once for each run.
Sun, Nov 1, 6:01 PM

Oct 29 2020

nate awarded Senior IAM Policy Wizard to recipient: bob.
Oct 29 2020, 4:25 PM
nate created Senior IAM Policy Wizard.
Oct 29 2020, 4:24 PM

Oct 28 2020

nate accepted D4925: [dask] allow connecting to existing scheduler.
Oct 28 2020, 5:18 PM
nate accepted D4938: re-run parse_spark_config.
Oct 28 2020, 5:18 PM
nate accepted D4932: fix make dev_install on py2.
Oct 28 2020, 1:36 AM

Oct 23 2020

nate added a comment to D4883: Break integration tests out into separate Python file.

ah, also I think we need to move the test image builds to this pipeline, from pipeline.py - see test_image_depends_fn

Oct 23 2020, 7:31 PM
nate added a comment to D4883: Break integration tests out into separate Python file.

Great! Adding @catherinewu also.

Oct 23 2020, 7:24 PM
nate added a reviewer for D4883: Break integration tests out into separate Python file: catherinewu.
Oct 23 2020, 7:23 PM

Oct 14 2020

nate accepted D4777: fix formatting for configsource.
Oct 14 2020, 9:15 PM
nate accepted D4640: publish lakehouse module.
Oct 14 2020, 9:15 PM

Oct 7 2020

nate accepted D4715: Revert "Revert "Add retries to dagster-postgres connections"".
Oct 7 2020, 10:22 PM
nate closed D4704: Revert "Add retries to dagster-postgres connections".
Oct 7 2020, 7:03 PM
nate committed R1:f9bed241cece: Revert "Add retries to dagster-postgres connections" (authored by nate).
Revert "Add retries to dagster-postgres connections"
Oct 7 2020, 7:03 PM
nate requested review of D4704: Revert "Add retries to dagster-postgres connections".
Oct 7 2020, 6:40 PM

Oct 6 2020

nate added a comment to D4449: Add retries to dagster-postgres connections.

@alangenfeld damn - worth reverting while we revise to hold onto connections? or just fix forward?

Oct 6 2020, 11:17 PM

Oct 5 2020

nate closed D4664: Fix PipelineRunsFilter typos.
Oct 5 2020, 7:40 PM
nate committed R1:b998dfed1027: Fix PipelineRunsFilter typos (authored by nate).
Fix PipelineRunsFilter typos
Oct 5 2020, 7:40 PM
nate accepted D4666: Don't check DAGSTER_REPO environment variable when building docker images.

Cool, thanks for this - shouldn't cause any other issues AFAIK

Oct 5 2020, 3:34 PM

Oct 3 2020

nate requested review of D4664: Fix PipelineRunsFilter typos.
Oct 3 2020, 10:27 PM

Oct 2 2020

nate accepted D4651: Add missing packaging dependency in dagster-aws.
Oct 2 2020, 7:32 PM
nate accepted D4629: Add dagster-version arg to docker release commands.

ah, great! once landed could you update the documentation also?

Oct 2 2020, 3:34 PM

Oct 1 2020

nate accepted D4614: Revert "publish lakehouse module".
Oct 1 2020, 7:42 PM
nate published D4614: Revert "publish lakehouse module" for review.
Oct 1 2020, 7:42 PM

Sep 29 2020

nate updated subscribers of D4538: Write non-error execution logs to stdout instead of stderr, only surface stderr from child processes in host processes like the scheduler.

hmm I haven't worked on this part of the system much, but the proposal sounds reasonable; would love to hear from @max and @prha when he's back

Sep 29 2020, 7:26 PM
nate accepted D4585: fixed gcp import for apidocs.
Sep 29 2020, 7:20 PM

Sep 28 2020

nate added a comment to D4565: Add more info to pipeline run status checks.

any thoughts on how to test this? Would love to get some test coverage before merging

Sep 28 2020, 3:46 PM

Sep 26 2020

nate requested review of D4565: Add more info to pipeline run status checks.
Sep 26 2020, 2:46 AM
nate committed R1:a256cb43cde0: Add more info to exception when pipeline run is in the wrong state (authored by nate).
Add more info to exception when pipeline run is in the wrong state
Sep 26 2020, 2:16 AM
nate closed D4560: Add more info to exception when pipeline run is in the wrong state.
Sep 26 2020, 2:16 AM
nate added a comment to D4560: Add more info to exception when pipeline run is in the wrong state.

yeah, let me do those separately, I'm less confident in being able to write good test for those—but will put up a follow-up diff

Sep 26 2020, 2:15 AM
nate accepted D4555: Update BK integration image.
Sep 26 2020, 1:57 AM
nate accepted D4561: update bk unit image.
Sep 26 2020, 1:57 AM
nate accepted D4550: publish lakehouse module.
Sep 26 2020, 12:28 AM

Sep 25 2020

nate requested review of D4560: Add more info to exception when pipeline run is in the wrong state.
Sep 25 2020, 10:00 PM
nate accepted D4544: Move Helm.
Sep 25 2020, 7:16 PM
nate committed R1:6cdb258f3e7b: Add retries to S3 object store client (authored by nate).
Add retries to S3 object store client
Sep 25 2020, 5:49 PM
nate closed D4487: Add retries to S3 object store client.
Sep 25 2020, 5:48 PM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 25 2020, 5:33 PM
nate accepted D4548: [helm] celery 5.0.0 fixes.
Sep 25 2020, 5:02 PM
nate updated the diff for D4487: Add retries to S3 object store client.

rebase

Sep 25 2020, 3:08 AM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 25 2020, 3:03 AM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 25 2020, 2:09 AM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 25 2020, 2:08 AM
nate added inline comments to D4487: Add retries to S3 object store client.
Sep 25 2020, 2:08 AM
nate committed R1:4299fa61f448: Fix build_js.sh for paths with spaces (authored by nate).
Fix build_js.sh for paths with spaces
Sep 25 2020, 1:55 AM
nate closed D4543: Fix build_js.sh for paths with spaces.
Sep 25 2020, 1:55 AM

Sep 24 2020

nate requested review of D4543: Fix build_js.sh for paths with spaces.
Sep 24 2020, 11:52 PM
nate accepted D4527: add ability to specify custom celery liveness probe.

one nit, otherwise lgtm

Sep 24 2020, 6:04 PM

Sep 23 2020

nate accepted D4484: Streamline console log messages.
Sep 23 2020, 10:08 PM
nate accepted D4507: BK: wait for celery to init.

I think we should still do this regardless, but can you also take a look at using PG for queuing and dropping RabbitMQ entirely?

Sep 23 2020, 3:15 AM

Sep 21 2020

nate updated the test plan for D4487: Add retries to S3 object store client.
Sep 21 2020, 10:57 PM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 21 2020, 7:52 PM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 21 2020, 7:28 PM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 21 2020, 4:42 PM
nate added a comment to D4367: Celery run queue.

left a few comments, otherwise lgtm - will let @catherinewu have the honors of final sign-off

Sep 21 2020, 3:34 PM

Sep 19 2020

nate updated the diff for D4487: Add retries to S3 object store client.

rebase

Sep 19 2020, 8:36 PM
nate updated the diff for D4487: Add retries to S3 object store client.

up

Sep 19 2020, 4:16 PM
nate added a comment to D4487: Add retries to S3 object store client.

I don't know how best to test this, but see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html

Sep 19 2020, 3:38 AM
nate requested review of D4487: Add retries to S3 object store client.
Sep 19 2020, 3:35 AM

Sep 17 2020

nate committed R1:44418d67da1b: Add retries to dagster-postgres connections (authored by nate).
Add retries to dagster-postgres connections
Sep 17 2020, 10:14 PM
nate closed D4449: Add retries to dagster-postgres connections.
Sep 17 2020, 10:14 PM
nate updated the diff for D4449: Add retries to dagster-postgres connections.

up

Sep 17 2020, 3:50 PM
nate added inline comments to D4449: Add retries to dagster-postgres connections.
Sep 17 2020, 3:48 PM
nate added reviewers for D4449: Add retries to dagster-postgres connections: catherinewu, johann.
Sep 17 2020, 3:02 AM

Sep 16 2020

nate added inline comments to D4449: Add retries to dagster-postgres connections.
Sep 16 2020, 10:38 PM
nate added reviewers for D4449: Add retries to dagster-postgres connections: alangenfeld, dgibson, max.
Sep 16 2020, 10:28 PM
nate updated the diff for D4449: Add retries to dagster-postgres connections.

up

Sep 16 2020, 10:24 PM
nate updated the diff for D4449: Add retries to dagster-postgres connections.

up

Sep 16 2020, 10:08 PM
nate updated the diff for D4449: Add retries to dagster-postgres connections.

up

Sep 16 2020, 9:56 PM
nate updated the summary of D4449: Add retries to dagster-postgres connections.
Sep 16 2020, 9:54 PM
nate updated the diff for D4449: Add retries to dagster-postgres connections.

up

Sep 16 2020, 9:40 PM
nate updated the diff for D4449: Add retries to dagster-postgres connections.

up

Sep 16 2020, 9:39 PM
nate requested review of D4449: Add retries to dagster-postgres connections.
Sep 16 2020, 9:20 PM

Sep 15 2020

nate accepted D4429: log link to databricks ui after launching job.
Sep 15 2020, 11:19 PM
nate accepted D4411: get spark-s3 type storage plugin working with configured storages.
Sep 15 2020, 8:26 PM

Sep 14 2020

nate accepted D4423: apidoc for emr_pyspark_step_launcher.
Sep 14 2020, 7:00 PM

Sep 11 2020

nate accepted D4403: default name for create_databricks_job_solid.
Sep 11 2020, 6:18 PM
nate committed R1:dfce9a07ab1d: Make Scala event processing less fancy (authored by nate).
Make Scala event processing less fancy
Sep 11 2020, 4:00 PM
nate closed D4399: Make Scala event processing less fancy.
Sep 11 2020, 4:00 PM
nate requested review of D4399: Make Scala event processing less fancy.
Sep 11 2020, 4:49 AM
nate accepted D4395: fix databricks api call.
Sep 11 2020, 1:43 AM

Sep 10 2020

nate accepted D4387: add tests for pyspark resource.
Sep 10 2020, 7:04 PM

Aug 27 2020

nate accepted D3972: Use configured in emr pyspark example.

nice! this is sweet

Aug 27 2020, 11:40 PM

Aug 25 2020

nate accepted D4238: Drop support for Python 2 in dagster-pyspark.
Aug 25 2020, 10:19 PM

Aug 21 2020

nate accepted D4163: Turn on auto-formatting for string literals in black.

after a long stint of meticulously switching double quotes to single wherever I found them, given the abstentions, I'm happy to be the one to go ahead and greenlight this. Getting tooling support is super attractive

Aug 21 2020, 8:53 PM