Page MenuHomeElementl

decouple api docs dependencies from `make dev_install`
ClosedPublic

Authored by yuhan on Feb 2 2021, 12:26 AM.

Details

Reviewers
nate
Group Reviewers
Restricted Project
Commits
R1:7c66b0dc0450: decouple api docs dependencies from `make dev_install`
Summary

This diff decouple the docs requirements from the dev_install.
So, for example, dagster-airflow api docs can be built properly (it requires both dagster-airflow and airlfow but we can't install airflow in dev_install as discussed in D6206)

We will use pip install -r docs-requirements.txt instead of make dev_install to install docs dependency

I updated the "Publish the docs" section too to reflect the change.

In the long term, the plan with the new site is to use a docker image to release the docs so it won't be affected by individual local virtualenv.

Test Plan

in a fresh virtualenv

~/dev/dagster yuhan/docs-auto-build*
$ test-docs-auto-build $ pip install -r docs-requirements.txt -qqq
astroid 2.4.2 has requirement lazy-object-proxy==1.4.*, but you'll have lazy-object-proxy 1.5.2 which is incompatible.
tox 3.14.2 has requirement importlib-metadata<2,>=1.1.0; python_version < "3.8", but you'll have importlib-metadata 3.4.0 which is incompatible.
marshmallow-sqlalchemy 0.24.1 has requirement marshmallow>=3.0.0, but you'll have marshmallow 2.21.0 which is incompatible.
flask-jwt-extended 3.25.0 has requirement PyJWT<2.0,>=1.6.4, but you'll have pyjwt 2.0.1 which is incompatible.
apache-airflow 1.10.10 has requirement attrs~=19.3, but you'll have attrs 20.3.0 which is incompatible.
apache-airflow 1.10.10 has requirement croniter<0.4,>=0.3.17, but you'll have croniter 1.0.6 which is incompatible.
apache-airflow 1.10.10 has requirement jinja2<2.11.0,>=2.10.1, but you'll have jinja2 2.11.3 which is incompatible.
apache-airflow 1.10.10 has requirement pandas<1.0.0,>=0.17.1, but you'll have pandas 1.2.1 which is incompatible.
apache-airflow 1.10.10 has requirement werkzeug<1.0.0, but you'll have werkzeug 1.0.1 which is incompatible.
botocore 1.12.253 has requirement docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.
botocore 1.12.253 has requirement urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 1.26.3 which is incompatible.
databricks-cli 0.12.2 has requirement tenacity>=6.2.0, but you'll have tenacity 4.12.0 which is incompatible.
dagster-pandas 0.10.2 has requirement pandas<=1.1.4, but you'll have pandas 1.2.1 which is incompatible.
great-expectations 0.12.10 has requirement requests<2.24,>=2.20, but you'll have requests 2.25.1 which is incompatible.
snowflake-connector-python 2.3.10 has requirement pytz<2021.0, but you'll have pytz 2021.1 which is incompatible.
twilio 6.51.1 has requirement PyJWT==1.7.1, but you'll have pyjwt 2.0.1 which is incompatible.

although there are many conflicts, it doesn't affect the Sphinx build process. We still get all related api docs generated correctly - i think it's because those incompatible packages weren't used when generating the api references.

make buildnext && cd next && yarn && yarn dev: localhost has no empty api page

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

yuhan edited the test plan for this revision. (Show Details)
yuhan edited the test plan for this revision. (Show Details)
yuhan requested review of this revision.Feb 2 2021, 12:46 AM
nate added inline comments.
.buildkite/dagster-buildkite/dagster_buildkite/steps/docs.py
32–33

installing automation should probably come after installing docs-requirements.txt now, since it depends on dagster. You could even move this requirement into docs-requirements.txt?

This revision is now accepted and ready to land.Feb 2 2021, 2:30 AM

see one comment, otherwise LGTM!

move ./python_modules/automation to docs-requirements.txt