Page MenuHomeElementl

RFC: remove the airline demo
Needs RevisionPublic

Authored by sandyryza on Apr 9 2021, 11:27 PM.

Details

Summary

The main reason to do this is that it's unsupported. It's broken in a few different ways (https://github.com/dagster-io/dagster/issues/3577), and we don't have plans to fix it or to make sure it keeps working if we were to fix it. This communicates immaturity and wastes users' time.

I think the content is overall quite good If we do want to keep it around, here are the things that I think we'd need to do to make it an example I'd feel confident about pointing people to:

  • Anchor it in some description of what it's trying to accomplish. Is the goal to build some report that people would reference regularly? An ad-hoc data analysis that can be re-run manually when new facts come to light? Feed an application?
  • Fix the issues referenced above.
  • Use Python type annotations instead of make_python_type_usable_as_dagster_type etc.
  • Deal with the commented-out materialization_strategy_output_types in sql_solid.
  • It uses solids in places where I think regular Python functions would be wiser, for example in s3_to_dw_table and s3_to_df. These incur unnecessary materializations and step startup overheads.
Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
rm-airline (branched from master)
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 9 2021, 11:28 PM
Harbormaster failed remote builds in B28675: Diff 35197!
sandyryza retitled this revision from remove the airline demo to RFC: remove the airline demo.
sandyryza edited the summary of this revision. (Show Details)

i vote for yes, lets remove the airline demo.

i think it's very reasonable to replace it with the hacker news demo, because

  1. the hacker news demo is a lot lightweight for users to follow and for us to maintain, and i don't see the airline demo showcases significantly different use cases.
  2. we don't really mention it anywhere on the docs and haven't got users asking for it for a while.
  3. overhauling it to use the up-to-date features and maintaining two demos going forward seems to be a non-trivial effort.

if we ever want to bring it back, say in the future we're ready to maintain more than one large demo (or #2451), we can always refresh it and add it back.

I'd like to have a nice dagstermill example in the repo, and this was the best one. Other than that I support deleting.

However, I think the we should have hacker news pipeline in the public repo before committing.

Can we just add some notebooks to hacker news?

Q management. lets do it after moving hacker news to public repo

This revision now requires changes to proceed.Jul 7 2021, 9:15 PM