Page MenuHomeElementl

RFC: remove the airline demo
Needs RevisionPublic

Authored by sandyryza on Apr 9 2021, 11:27 PM.

Details

Summary

The main reason to do this is that it's unsupported. It's broken in a few different ways (https://github.com/dagster-io/dagster/issues/3577), and we don't have plans to fix it or to make sure it keeps working if we were to fix it. This communicates immaturity and wastes users' time.

I think the content is overall quite good If we do want to keep it around, here are the things that I think we'd need to do to make it an example I'd feel confident about pointing people to:

  • Anchor it in some description of what it's trying to accomplish. Is the goal to build some report that people would reference regularly? An ad-hoc data analysis that can be re-run manually when new facts come to light? Feed an application?
  • Fix the issues referenced above.
  • Use Python type annotations instead of make_python_type_usable_as_dagster_type etc.
  • Deal with the commented-out materialization_strategy_output_types in sql_solid.
  • It uses solids in places where I think regular Python functions would be wiser, for example in s3_to_dw_table and s3_to_df. These incur unnecessary materializations and step startup overheads.
Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
rm-airline (branched from master)
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 9 2021, 11:28 PM
Harbormaster failed remote builds in B28675: Diff 35197!
sandyryza retitled this revision from remove the airline demo to RFC: remove the airline demo.
sandyryza edited the summary of this revision. (Show Details)

i vote for yes, lets remove the airline demo.

i think it's very reasonable to replace it with the hacker news demo, because

  1. the hacker news demo is a lot lightweight for users to follow and for us to maintain, and i don't see the airline demo showcases significantly different use cases.
  2. we don't really mention it anywhere on the docs and haven't got users asking for it for a while.
  3. overhauling it to use the up-to-date features and maintaining two demos going forward seems to be a non-trivial effort.

if we ever want to bring it back, say in the future we're ready to maintain more than one large demo (or #2451), we can always refresh it and add it back.

I'd like to have a nice dagstermill example in the repo, and this was the best one. Other than that I support deleting.

However, I think the we should have hacker news pipeline in the public repo before committing.

Can we just add some notebooks to hacker news?

yuhan requested changes to this revision.Wed, Jul 7, 9:15 PM

Q management. lets do it after moving hacker news to public repo

This revision now requires changes to proceed.Wed, Jul 7, 9:15 PM