Page MenuHomeElementl

Hackernews demo in public repo
ClosedPublic

Authored by owen on Jun 7 2021, 10:23 PM.

Details

Summary

This is just an initial commit to see if I can get tests passing. Still need to remove some things here and there to simplify this a bit (ex: the dynamic download pipeline stuff)

Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 7 2021, 11:04 PM
Harbormaster failed remote builds in B31779: Diff 39138!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 8 2021, 12:01 AM
Harbormaster failed remote builds in B31784: Diff 39145!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 8 2021, 5:57 PM
Harbormaster failed remote builds in B31807: Diff 39171!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 8 2021, 6:33 PM
Harbormaster failed remote builds in B31810: Diff 39177!
Harbormaster returned this revision to the author for changes because remote builds failed.Jun 8 2021, 7:17 PM
Harbormaster failed remote builds in B31813: Diff 39180!
  • removed some fluff
  • removed even more fluff
  • added some explanations
owen requested review of this revision.Jun 9 2021, 12:07 AM
  • added some explanations

Nice!

After this goes in, what will the procedure be for making changes to the pipelines? Will there be some intermediate period during which we need to replicate changes across both the public repo and the demo repo? Not the end of the world, but ultimately will be pretty painful. Do we have a plan for making the demo depend on the public repo version of the pipeline?

examples/hacker_news/hacker_news/resources/bq_resource.py
1 ↗(On Diff #39224)

Is this needed anymore?

examples/hacker_news/hacker_news/resources/parquet_pointer.py
5

Preferable:

from typing import NamedTuple:

class ParquetPointer(NamedTuple):
    path: str
    schema: str
This revision is now accepted and ready to land.Jun 9 2021, 5:31 PM

Nice!

After this goes in, what will the procedure be for making changes to the pipelines? Will there be some intermediate period during which we need to replicate changes across both the public repo and the demo repo? Not the end of the world, but ultimately will be pretty painful. Do we have a plan for making the demo depend on the public repo version of the pipeline?

re: the making changes thing, I do foresee some time period where we'll have to manually replicate the changes between these two (ex: once @cdecarolis lands https://dagster.phacility.com/D8236, I'll handle porting that change over here for him). There's a bit of work that will be necessary to actually link these together, because I threw out a fair number of things that only made sense in the internal repo, like HNAPISubsampleClient, in order to make the repo a bit clearer for someone poking around in it.

My hope is that these changes won't be super frequent in the short term, so that the pain will be fairly limited.

  • added some explanations
  • changed ParquetPointer to NamedTuple
This revision was automatically updated to reflect the committed changes.