Page MenuHomePhabricator
Feed Advanced Search

Today

schrockn requested changes to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.

req'ing changes mostly based on the comment about the __class__ key

Fri, Jul 10, 1:32 AM
schrockn committed R1:18db7766c3a0: s/ditionary/dictionary (authored by schrockn).
s/ditionary/dictionary
Fri, Jul 10, 1:20 AM
schrockn closed D3830: s/ditionary/dictionary.
Fri, Jul 10, 1:20 AM
schrockn accepted D3825: (automation 1/) Move git utils to automation.
Fri, Jul 10, 1:19 AM

Yesterday

schrockn requested review of D3830: s/ditionary/dictionary.
Thu, Jul 9, 11:02 PM
schrockn requested review of D3829: RFC: mapped type convention.
Thu, Jul 9, 10:59 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Thu, Jul 9, 10:32 PM
schrockn accepted D3806: add SystemExecutionContext.

makes sense

Thu, Jul 9, 7:44 PM
schrockn requested review of D3823: (refactor-generator-exit-3) Another conversion.
Thu, Jul 9, 7:06 PM
schrockn accepted D3767: dont progress skips past a fan-in dependency.

per discussion on call let's move ahead with this

Thu, Jul 9, 7:03 PM
schrockn requested review of D3821: (refactor-generator-exit-2) Another generator exit conversion.
Thu, Jul 9, 7:03 PM
schrockn added inline comments to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.
Thu, Jul 9, 6:56 PM
schrockn accepted D3567: [docs] Add code example for intro page.

cool thx. probably update diff description in indicated reduced scope of this diff

Thu, Jul 9, 6:52 PM
schrockn added a comment to D3787: RFC: Proposed pattern for GeneratorExit shenanigans.

Look at https://dagster.phacility.com/D3821 and https://dagster.phacility.com/D3823 to see if we should move forward with this

Thu, Jul 9, 6:51 PM
schrockn added a comment to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.

also having the persistence decorator I think it really value. right now it's impossible to know by inspection what serdes classes are persisted versus not. therefore if you add a param or field or whatever, you have to inspect the whole system and make sure that it is not persisted anywhere, else you risk breakage. this makes it much more clear what is actually persisted.

Thu, Jul 9, 5:54 PM
schrockn added a comment to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.

@alangenfeld it shuffles around complexity quite a bit but what I do think it makes much, much clear is how the thing is supposed to be used when it is built by our business logic code as opposed to reconstructing from persistence.

Thu, Jul 9, 5:52 PM
schrockn requested changes to D3567: [docs] Add code example for intro page.

I think this is a good starting point.

Thu, Jul 9, 5:51 PM
schrockn added inline comments to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.
Thu, Jul 9, 5:40 PM
schrockn added a comment to D3817: Pass in DagsterInstance to RunLaunchers before launch_run.

Oh I didn't consider that. Hmm. Is there good way we can warn in the interim and then do a cutover?

Thu, Jul 9, 5:33 PM
schrockn updated the diff for D3802: Unit test buildkite image.

undoing -U

Thu, Jul 9, 5:26 PM
schrockn added a comment to D3802: Unit test buildkite image.

One aspect I do find confusing is why this hasn't been caught on the existing integration images. This change changes nothing about that behavior. Presumably having all modules pre-installed should have had the same issues as before, but we have still been picking up ecosystem changes. I would rather (a) understand what is happening and (b) decide on a policy change across our entire infrastructure rather than just add it here only on the built test image.

Thu, Jul 9, 5:25 PM
schrockn accepted D3817: Pass in DagsterInstance to RunLaunchers before launch_run.

despite that this still seems like a sensible incremental change

Thu, Jul 9, 4:57 PM
schrockn added inline comments to D3817: Pass in DagsterInstance to RunLaunchers before launch_run.
Thu, Jul 9, 4:56 PM
schrockn added a comment to D3802: Unit test buildkite image.
  • What are the concerns about -U seems like what is being asked for
  • Re: proxy network I'm not convinced this is the biggest issue. You should still have to do all the local compute and disk work to construct the environment which I think is the bulk of the expense.
  • The tradeoff between having a stable internal build versus discover of ecosystem issues is an interesting one. I generally bias towards us not getting thrashed the moment that anything breaks in the entire ecosystem. Frequently these problems get resolved quickly anyways and we get thrashed when it would be been resolved anyways. However I'm open to persuasion on this point and am curious what other folks think.
Thu, Jul 9, 4:43 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Thu, Jul 9, 4:40 PM
schrockn added inline comments to D3802: Unit test buildkite image.
Thu, Jul 9, 4:38 PM
schrockn added inline comments to D3802: Unit test buildkite image.
Thu, Jul 9, 4:37 PM
schrockn requested changes to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.

q mgmt

Thu, Jul 9, 4:22 PM
schrockn accepted D3817: Pass in DagsterInstance to RunLaunchers before launch_run.

excellent change change. should have been like this the whole time

Thu, Jul 9, 4:21 PM
schrockn added a reviewer for D3802: Unit test buildkite image: johann.
Thu, Jul 9, 4:02 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Thu, Jul 9, 2:09 AM

Wed, Jul 8

schrockn updated the diff for D3802: Unit test buildkite image.

up

Wed, Jul 8, 11:35 PM
schrockn added a comment to D3757: Reliable snapshot generation and update snapshot images.

For posterity:

Wed, Jul 8, 10:46 PM
schrockn added a reviewer for D3802: Unit test buildkite image: max.
Wed, Jul 8, 9:34 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Wed, Jul 8, 9:30 PM
schrockn updated the summary of D3802: Unit test buildkite image.
Wed, Jul 8, 9:27 PM
schrockn updated the test plan for D3802: Unit test buildkite image.
Wed, Jul 8, 9:26 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Wed, Jul 8, 9:25 PM
schrockn added a comment to D3802: Unit test buildkite image.

Before:

Wed, Jul 8, 9:21 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Wed, Jul 8, 9:03 PM
schrockn added inline comments to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.
Wed, Jul 8, 9:00 PM
schrockn added a comment to D3809: RFC: add PersistentTuple, to add layer of indirection from serialization.

I think it would be useful to show what PipelineRun would look like with this

Wed, Jul 8, 8:59 PM
schrockn updated the diff for D3802: Unit test buildkite image.

up

Wed, Jul 8, 7:19 PM
schrockn updated the diff for D2155: Warm the BK cluster.

up

Wed, Jul 8, 1:33 PM
schrockn added a comment to D3797: move API run launcher to store psutil.Process objects instead of POpen objects.

Yeah to be clear it doesn't make that that that would be the issue, since the integration image includes an explicit install of gcc. It's just something that has changed.

Wed, Jul 8, 1:32 PM
schrockn requested review of D3802: Unit test buildkite image.
Wed, Jul 8, 4:22 AM
schrockn updated the diff for D2155: Warm the BK cluster.

up

Wed, Jul 8, 4:12 AM
schrockn added a comment to D3797: move API run launcher to store psutil.Process objects instead of POpen objects.

I updated the integration image today, so this could be related to that somehow.

Wed, Jul 8, 4:03 AM
schrockn committed R1:c3fc621d05fb: Coverage-specific docker image (authored by schrockn).
Coverage-specific docker image
Wed, Jul 8, 2:18 AM
schrockn closed D3800: Coverage-specific docker image.
Wed, Jul 8, 2:18 AM
schrockn updated the diff for D3800: Coverage-specific docker image.

up

Wed, Jul 8, 1:54 AM
schrockn updated the summary of D3800: Coverage-specific docker image.
Wed, Jul 8, 1:52 AM
schrockn requested review of D3800: Coverage-specific docker image.

Warm Start:

Wed, Jul 8, 1:52 AM
schrockn updated the diff for D2155: Warm the BK cluster.

up

Wed, Jul 8, 1:35 AM
schrockn committed R1:3ed7d55aa8b3: Reliable snapshot generation and update snapshot images (authored by schrockn).
Reliable snapshot generation and update snapshot images
Wed, Jul 8, 12:11 AM
schrockn closed D3757: Reliable snapshot generation and update snapshot images.
Wed, Jul 8, 12:11 AM

Tue, Jul 7

schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 11:53 PM
schrockn added inline comments to D3757: Reliable snapshot generation and update snapshot images.
Tue, Jul 7, 11:53 PM
schrockn added a comment to D3757: Reliable snapshot generation and update snapshot images.

Btw re workflows, step 1: base image build & push (local), should be quite rare. That contains stuff that doesn't change too often (e.g. sbt installation etc).

Tue, Jul 7, 11:44 PM
schrockn added a comment to D3779: launch dockerized postgres in a common/reusable way.

this is great. not a huge deal but i think a more natural home for this would be the dagster_test module which is for test infra like this. doesn't make sense to ship this code to production users

Tue, Jul 7, 11:32 PM
schrockn requested changes to D3767: dont progress skips past a fan-in dependency.
Tue, Jul 7, 10:55 PM
schrockn accepted D3781: Use `StringSource` for `executable_path` value in python env repo location config.
Tue, Jul 7, 10:50 PM
schrockn accepted D3782: Update `load_workspace_from_yaml_path` to take multiple yaml paths.
Tue, Jul 7, 10:48 PM
schrockn abandoned D3668: add cli argument.
Tue, Jul 7, 10:46 PM
schrockn abandoned D3687: remove pandas and airflow.
Tue, Jul 7, 10:46 PM
schrockn abandoned D3690: python images.
Tue, Jul 7, 10:46 PM
schrockn accepted D3796: [easy] Update cli tests to use workspace.yaml.
Tue, Jul 7, 10:43 PM
schrockn accepted D3794: Fix a Gaant chart layout issue with unstarted steps appearing at x=0.

Oh great. This bug has been annoying me for awhile :-) I think Prezi noted it too so maybe add an entry to the changelog?

Tue, Jul 7, 10:38 PM
schrockn accepted D3795: final input_hydration/output_materialization cleanups.
Tue, Jul 7, 10:38 PM
schrockn requested changes to D3784: RFC: Allow composite solids to yield outputs from a generator.

Yeah this gives me the heebie jeebies as well. I think it's because we are overlaoding the Output class to mean something subtly different which could in fact cause more confusion. Will people expect AssetMaterialization or ExpectationResult to work? I think it just might cause further confusion and make people think the @composite_solids actually do some computation at runtime rather than at construction time.

Tue, Jul 7, 10:37 PM
schrockn committed R1:a614b9299de8: Complete putting DagsterKubernetesClient under test (authored by schrockn).
Complete putting DagsterKubernetesClient under test
Tue, Jul 7, 10:08 PM
schrockn closed D3764: Complete putting DagsterKubernetesClient under test.
Tue, Jul 7, 10:08 PM
schrockn retitled D3757: Reliable snapshot generation and update snapshot images from Reliable snapshot generation to Reliable snapshot generation and update snapshot images.
Tue, Jul 7, 9:43 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 9:42 PM
schrockn added a reviewer for D3757: Reliable snapshot generation and update snapshot images: yuhan.
Tue, Jul 7, 9:39 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 9:33 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 9:32 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 9:21 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 9:19 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:58 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:49 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:46 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:43 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:39 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:35 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:31 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:28 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:25 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:22 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:17 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:14 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:12 PM
schrockn removed reviewers for D3757: Reliable snapshot generation and update snapshot images: nate, catherinewu, yuhan.
Tue, Jul 7, 8:05 PM
schrockn updated the diff for D3757: Reliable snapshot generation and update snapshot images.

up

Tue, Jul 7, 8:02 PM
schrockn requested review of D3787: RFC: Proposed pattern for GeneratorExit shenanigans.
Tue, Jul 7, 7:58 PM
schrockn updated the diff for D3764: Complete putting DagsterKubernetesClient under test.

up

Tue, Jul 7, 7:30 PM
schrockn updated the diff for D2155: Warm the BK cluster.

up

Tue, Jul 7, 7:29 PM
schrockn accepted D3793: fix input_hydration_config backcompat issue.

is this worthy of a release @sandyryza ?

Tue, Jul 7, 7:18 PM
schrockn committed R1:59232fa67bc5: Always cleanup cluster in BK (authored by schrockn).
Always cleanup cluster in BK
Tue, Jul 7, 7:08 PM
schrockn closed D3790: Always cleanup cluster in BK.
Tue, Jul 7, 7:08 PM