Page MenuHomeElementl

Adds content for init guide (new-repo CLI)
ClosedPublic

Authored by bob on Feb 25 2021, 6:46 PM.

Details

Summary

Adds content for the Getting Started -> Create a New Project in the new documentation site for 0.11.0.

Also, Next seems to generate files in docs/out/, which bloats the git history. So, docs/out/ was added to a .gitignore.

Test Plan

buildkite

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 25 2021, 7:05 PM
Harbormaster failed remote builds in B26445: Diff 32313!

I think this will be a really helpful addition.

I think we should be careful to indicate these are a set of conventions, but that this structure isn't required for Dagster to work. Many people who are coming to Dagster will have existing data processing code, and it might not be convenient for them to restructure how that code is organized in their git repository. This relates to my comment above about avoiding structuring as a linear journey.

docs/next/content/getting-started/init.mdx
8 ↗(On Diff #32313)

This is a small thing, but I'd consider taking out the "In the previous section" - I don't think we should necessarily structure "Getting Started" as a linear journey in the same way we do with the tutorial. I.e. users shouldn't feel like they need to follow all the steps to start using Dagster.

13 ↗(On Diff #32313)

While I think it would be cool if our tutorial taught people how to be effective data engineers, I don't think it does that right now.

24 ↗(On Diff #32313)

It seems like we use the term "repository" here to mean a subtly different third thing from the concepts above, which is something akin to "folder structure containing a set of source files which happen to contain a single repository definition". I worry that might cause confusion with the meaning of repository inside Dagster.

To draw a comparison that might make this more concrete, I assume we wouldn't feel comfortable using the word "pipeline" to refer to a set of source files that happen to contain a pipeline definition.

An alternative could be to call it something like "project skeleton".

68 ↗(On Diff #32313)

Maybe change to "to find where to load your pipelines, schedules, and sensors from".

Rebase on 11-docs branch.

Harbormaster returned this revision to the author for changes because remote builds failed.Mar 2 2021, 6:25 PM
Harbormaster failed remote builds in B26673: Diff 32600!
bob added inline comments.
docs/next/content/getting-started/init.mdx
8 ↗(On Diff #32313)

"I don't think we should necessarily structure "Getting Started" as a linear journey"

Gotcha. Makes a lot of sense

24 ↗(On Diff #32313)

"It seems like we use the term "repository" here to mean a subtly different third thing... I worry that might cause confusion with the meaning of repository inside Dagster."

Thanks for the feedback. I was actually thinking the same things when I wrote this out the first time.

An alternative could be to call it something like "project skeleton".

"Project skeleton" makes a lot of sense to me too, and I think it's better than overloading the term "repository" again. I was worried that readers might think that "project" is another Dagster concept 🙄

bob marked an inline comment as done.

Edits after feedback #1:

  • Remove "in the previous section", because the Getting Started guides are not cumulative.
  • Change wording when describing the generated "project skeleton", so as to avoid overloading the term "repository".
  • Adds an explanation and link for repository locations.
  • Adds detail that workspace.yaml is used to find repository locations, from which Dagster will load pipelines, etc.
Harbormaster returned this revision to the author for changes because remote builds failed.Mar 2 2021, 6:51 PM
Harbormaster failed remote builds in B26677: Diff 32604!

Adds whitespace in Markdown, as recommended by make snapshot.

this is looking pretty good!

left some comments regarding the narrative.

also, shall we rename the url to be "create-a-new-dagster-project" more something more explicit rather than init.mdx - that was just a placeholder name i put

docs/next/content/getting-started/init.mdx
3 ↗(On Diff #32612)

let's add a description here. maybe something like "Dagster provides a convenient CLI to easily start a Dagster project." - just a suggestion, up to you to tune the wording :)

8–9 ↗(On Diff #32612)

could call out the things before a user dives in:

  1. this is not required to get started
  2. some people may not need it. it's just the easiest to start a Dagster project
bob marked 3 inline comments as done.

Edits after feedback #2:

  • Change URL path from /getting-started/init to /getting-started/create-new-project.
  • Adds a description to frontmatter YAML.
  • In 1st paragraph, add a note that the project skeleton structure is not the only way.

lg2m. thank you for putting this together. let's land it!

we may want to do another pass as we get more feedback about the cli and the docs from the team :)

This revision is now accepted and ready to land.Mar 3 2021, 5:26 PM

Fix internal links.

Note: There currently does not seem to be an 11-docs page for Dagster daemon or Dagit. Once those pages are created, the links should be added to the /getting-started/create-new-project page.

Re-creates diff on master, since 11-docs has been merged onto master.

This revision was automatically updated to reflect the committed changes.