add markdown descriptions for dagster pandas dataframe types

Authored by themissinghlink on Jan 2 2020, 6:50 PM.


add markdown descriptions for dagster pandas dataframe types

This revision is tackling the addition of markdown descriptions for pandas dataframe types to enable useful understanding of data in dagit.

More specifically, this revision attempts to implement cases 1-3 of the following spec: https://hackmd.io/pqpg7dy9RjKlNa6CU8niVQ.

I am not handling 4 because that requires some changes to the base column api and I don't want to complicate this revision too much.


  • Added markdown description generation and hooked it into create_dagster_pandas_dataframe_type.
  • Added unit tests for create_dagster_pandas_dataframe_type description generation.
  • Established the concept of error_descriptions and markdown_descriptions so that error messaging during development was decoupled from the markdown syntax requirements in dagit.

Proof this works

Test Plan: unit

Reviewers: alangenfeld, max, schrockn

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D1753